Text summarization Phase 2 Evaluation 2

Phase 2 evaluation 2 of text summarization final year project, under professor U.A. Deshpande in collaboration with TCS, presentation.

Implementation of Seq2Seq model for auto text summarization using attention.

Phase 2 evaluation 1 presentation: https://speakerdeck.com/gautamabhishek46/text-summarization-phase-2-evaluation-1

Team:
Abhishek Gautam
Atharva Parwatkar
Sharvil Nagarkar

Professor in-charge: U. A. Deshpande
TCS Mentor : Dr. Sagar Sunkle

Transcript

1. Text Summarization Abhishek Gautam (BT15CSE002) Atharva Parwatkar (BT15CSE015) Sharvil Nagarkar (BT15CSE052) Under Prof. U.A.Deshpande
2. Recap • Last time we discussed about: ◦ Abstractive Summarisation ◦ Seq2Seq Neural Network Model ◦ Attention mechanism 2
3. Summarization using attention Model tries to rewrite a sentence in a different way. 3
4. Model Overview • Encoder-Decoder architecture ◦ Bidirectional LSTM as encoder ◦ Beam Search Decoder as decoder • Use of Bahdanau attention model 4
5. Model Architecture 5
6. Model Hyperparameters • Learning rate: 0.001 • Epochs: 10 • Beam width: 10 6
7. Some examples 7 S. No. Original text Transformed text 1 two more vietnamese professional football coaches have been suspended on suspicion of corruption or match-fixing in the country and abroad , officials said thursday . two more vietnamese football coaches suspended 2 the dollar inched higher against the euro thursday in thin trading on expectations the us economy saw robust job creation last month . dollar rises against euro 3 tourists from britain and ireland are flocking in ever greater numbers to the french riviera , helping make the region the second-most visited in france after paris , the area 's chamber of commerce said monday . < unk > tourists flock to france
8. Model Evaluation • For evaluation we are using ROUGE scores. • We have used ROUGE-1, ROUGE-2 and ROUGE-L metrics • ROUGE-1: Counts 1-grams • ROUGE-2: Counts 2-grams • ROUGE-L: Uses Longest Common Subsequence(LCS) 8
9. Evaluation Results 9 ROUGE-1 ROUGE-2 ROUGE-L Our Model 34.72 13.26 31.49 Pointer Generator Network 36.44 15.66 33.42
10. Some examples (CNN data) 10 Original text Transformed text James Best, best known for his portrayal of bumbling sheriff Rosco P. Coltrane on TV's "The Dukes of Hazzard," died Monday after a brief illness. He was 88. Best died in hospice in Hickory, North Carolina, of complications from pneumonia, said Steve Latshaw, a longtime friend and Hollywood colleague. influential political minister dies in death
11. Model Limitations • Currently not working very well with long texts • Very slow in training • Cannot understand out of vocabulary words 11
12. What next? • Hyperparameter tuning • Use of Luong attention • Reducing <unk> tokens 12
13. Hyperparameter Tuning • A hyperparameter is a parameter whose value is used to control the learning process. • E.g. learning rate, number of hidden layer neurons, number of epochs • Hyperparameter tuning is the problem of choosing a set of optimal hyperparameters for a learning algorithm. 13
14. Drawback of Bahdanau Attention • A major problem with Bahdanau attention model is that it tries to predict attention values for all hidden states. • This needs a lot of computations. • This is known as Global Attention. 14
15. Luong Attention • Luong attention solves these problems using the concept of Local attention. • It tries to predict attention around a window of hidden states. • Less computations mean less time for training. 15
16. Using Pointer Generator Network (PGN) • Current model is very heavy. • Generates <unk> tokens for out of vocabulary words. • Copying some part of input text can be helpful to generate summary. • It will reduce training time as well as <unk> tokens. • PGN can be said as a mixture of extractive and abstractive model. 16
17. Original Text a day after winning nigeria’s presidency, muhammadu buhari told cnn’s christiane amanpour that he plans to aggressively fight corruption that has long plagued nigeria and go after the root of the nation’s unrest. buhari said he’ll “rapidly give attention” to curbing violence in the northeast part of nigeria, where the terrorist group boko haram operates. by cooperating with neighboring nations chad, cameroon and niger, he said his administration is confident it will be able to thwart criminals and others contributing to nigeria’s instability. for the first time in nigeria’s history, the opposition defeated the ruling party in democratic elections. buhari defeated incumbent goodluck jonathan by about 2 million votes, according to nigeria’s independent national electoral commission. the win comes after a long history of military rule, coups and botched attempts at democracy in africa’s most populous nation. 17
18. With Attention UNK UNK says his administration is confident it will be able to destabilize nigeria’s economy. UNK says his administration is confident it will be able to thwart criminals and other nigerians. he says the country has long nigeria and nigeria’s economy. 18
19. With PGN muhammadu buhari says he plans to aggressively fight corruption in the northeast part of nigeria. he says he’ll “rapidly give attention” to curbing violence in the northeast part of nigeria. he says his administration is confident it will be able to thwart criminals 19
20. References • https://arxiv.org/pdf/1704.04368.pdf • https://arxiv.org/pdf/1509.00685.pdf • https://arxiv.org/pdf/1508.04025.pdf • https://www.coursera.org/learn/language-processing/lecture/RhxPO/get-to-the-point-summariz ation-with-pointer-generator-networks • https://machinelearningmastery.com/global-attention-for-encoder-decoder-recurrent-neural-net works/ • http://cnyah.com/2017/08/01/attention-variants/ 20
21. Thank you! 21