Slide 1

Slide 1 text

Text Summarization Abhishek Gautam (BT15CSE002) Atharva Parwatkar (BT15CSE015) Sharvil Nagarkar (BT15CSE052) Under Prof. U.A.Deshpande

Slide 2

Slide 2 text

Recap ● Last time we discussed about: ○ Abstractive Summarisation ○ Seq2Seq Neural Network Model ○ Attention mechanism 2

Slide 3

Slide 3 text

Summarization using attention Model tries to rewrite a sentence in a different way. 3

Slide 4

Slide 4 text

Model Overview ● Encoder-Decoder architecture ○ Bidirectional LSTM as encoder ○ Beam Search Decoder as decoder ● Use of Bahdanau attention model 4

Slide 5

Slide 5 text

Model Architecture 5

Slide 6

Slide 6 text

Model Hyperparameters ● Learning rate: 0.001 ● Epochs: 10 ● Beam width: 10 6

Slide 7

Slide 7 text

Some examples 7 S. No. Original text Transformed text 1 two more vietnamese professional football coaches have been suspended on suspicion of corruption or match-fixing in the country and abroad , officials said thursday . two more vietnamese football coaches suspended 2 the dollar inched higher against the euro thursday in thin trading on expectations the us economy saw robust job creation last month . dollar rises against euro 3 tourists from britain and ireland are flocking in ever greater numbers to the french riviera , helping make the region the second-most visited in france after paris , the area 's chamber of commerce said monday . < unk > tourists flock to france

Slide 8

Slide 8 text

Model Evaluation ● For evaluation we are using ROUGE scores. ● We have used ROUGE-1, ROUGE-2 and ROUGE-L metrics ● ROUGE-1: Counts 1-grams ● ROUGE-2: Counts 2-grams ● ROUGE-L: Uses Longest Common Subsequence(LCS) 8

Slide 9

Slide 9 text

Evaluation Results 9 ROUGE-1 ROUGE-2 ROUGE-L Our Model 34.72 13.26 31.49 Pointer Generator Network 36.44 15.66 33.42

Slide 10

Slide 10 text

Some examples (CNN data) 10 Original text Transformed text James Best, best known for his portrayal of bumbling sheriff Rosco P. Coltrane on TV's "The Dukes of Hazzard," died Monday after a brief illness. He was 88. Best died in hospice in Hickory, North Carolina, of complications from pneumonia, said Steve Latshaw, a longtime friend and Hollywood colleague. influential political minister dies in death

Slide 11

Slide 11 text

Model Limitations ● Currently not working very well with long texts ● Very slow in training ● Cannot understand out of vocabulary words 11

Slide 12

Slide 12 text

What next? ● Hyperparameter tuning ● Use of Luong attention ● Reducing tokens 12

Slide 13

Slide 13 text

Hyperparameter Tuning ● A hyperparameter is a parameter whose value is used to control the learning process. ● E.g. learning rate, number of hidden layer neurons, number of epochs ● Hyperparameter tuning is the problem of choosing a set of optimal hyperparameters for a learning algorithm. 13

Slide 14

Slide 14 text

Drawback of Bahdanau Attention ● A major problem with Bahdanau attention model is that it tries to predict attention values for all hidden states. ● This needs a lot of computations. ● This is known as Global Attention. 14

Slide 15

Slide 15 text

Luong Attention ● Luong attention solves these problems using the concept of Local attention. ● It tries to predict attention around a window of hidden states. ● Less computations mean less time for training. 15

Slide 16

Slide 16 text

Using Pointer Generator Network (PGN) ● Current model is very heavy. ● Generates tokens for out of vocabulary words. ● Copying some part of input text can be helpful to generate summary. ● It will reduce training time as well as tokens. ● PGN can be said as a mixture of extractive and abstractive model. 16

Slide 17

Slide 17 text

Original Text a day after winning nigeria’s presidency, muhammadu buhari told cnn’s christiane amanpour that he plans to aggressively fight corruption that has long plagued nigeria and go after the root of the nation’s unrest. buhari said he’ll “rapidly give attention” to curbing violence in the northeast part of nigeria, where the terrorist group boko haram operates. by cooperating with neighboring nations chad, cameroon and niger, he said his administration is confident it will be able to thwart criminals and others contributing to nigeria’s instability. for the first time in nigeria’s history, the opposition defeated the ruling party in democratic elections. buhari defeated incumbent goodluck jonathan by about 2 million votes, according to nigeria’s independent national electoral commission. the win comes after a long history of military rule, coups and botched attempts at democracy in africa’s most populous nation. 17

Slide 18

Slide 18 text

With Attention UNK UNK says his administration is confident it will be able to destabilize nigeria’s economy. UNK says his administration is confident it will be able to thwart criminals and other nigerians. he says the country has long nigeria and nigeria’s economy. 18

Slide 19

Slide 19 text

With PGN muhammadu buhari says he plans to aggressively fight corruption in the northeast part of nigeria. he says he’ll “rapidly give attention” to curbing violence in the northeast part of nigeria. he says his administration is confident it will be able to thwart criminals 19

Slide 20

Slide 20 text

References ● https://arxiv.org/pdf/1704.04368.pdf ● https://arxiv.org/pdf/1509.00685.pdf ● https://arxiv.org/pdf/1508.04025.pdf ● https://www.coursera.org/learn/language-processing/lecture/RhxPO/get-to-the-point-summariz ation-with-pointer-generator-networks ● https://machinelearningmastery.com/global-attention-for-encoder-decoder-recurrent-neural-net works/ ● http://cnyah.com/2017/08/01/attention-variants/ 20

Slide 21

Slide 21 text

Thank you! 21