Text summarization Phase 2 Evaluation 2

Phase 2 evaluation 2 of text summarization final year project, under professor U.A. Deshpande in collaboration with TCS, presentation.

Implementation of Seq2Seq model for auto text summarization using attention.

Phase 2 evaluation 1 presentation: https://speakerdeck.com/gautamabhishek46/text-summarization-phase-2-evaluation-1

Team:
Abhishek Gautam
Atharva Parwatkar
Sharvil Nagarkar

Professor in-charge: U. A. Deshpande
TCS Mentor : Dr. Sagar Sunkle

Transcript

1. Text Summarization
   Abhishek Gautam (BT15CSE002)
   Atharva Parwatkar (BT15CSE015)
   Sharvil Nagarkar (BT15CSE052)
   Under Prof. U.A.Deshpande
   

   View Slide

2. Recap
   ● Last time we discussed about:
   ○ Abstractive Summarisation
   ○ Seq2Seq Neural Network Model
   ○ Attention mechanism
   2
   

   View Slide

3. Summarization using attention
   Model tries to rewrite a sentence in a different way.
   3
   

   View Slide

4. Model Overview
   ● Encoder-Decoder architecture
   ○ Bidirectional LSTM as encoder
   ○ Beam Search Decoder as decoder
   ● Use of Bahdanau attention model
   4
   

   View Slide

5. Model Architecture
   5
   

   View Slide

6. Model Hyperparameters
   ● Learning rate: 0.001
   ● Epochs: 10
   ● Beam width: 10
   6
   

   View Slide

7. Some examples
   7
   S. No. Original text Transformed text
   1 two more vietnamese professional football coaches
   have been suspended on suspicion of corruption or
   match-fixing in the country and abroad , officials said
   thursday .
   two more vietnamese football coaches
   suspended
   2 the dollar inched higher against the euro thursday in
   thin trading on expectations the us economy saw
   robust job creation last month .
   dollar rises against euro
   3 tourists from britain and ireland are flocking in ever
   greater numbers to the french riviera , helping make
   the region the second-most visited in france after paris
   , the area 's chamber of commerce said monday .
   < unk > tourists flock to france
   

   View Slide

8. Model Evaluation
   ● For evaluation we are using ROUGE scores.
   ● We have used ROUGE-1, ROUGE-2 and ROUGE-L metrics
   ● ROUGE-1: Counts 1-grams
   ● ROUGE-2: Counts 2-grams
   ● ROUGE-L: Uses Longest Common Subsequence(LCS)
   8
   

   View Slide

9. Evaluation Results
   9
   ROUGE-1 ROUGE-2 ROUGE-L
   Our Model 34.72 13.26 31.49
   Pointer Generator
   Network
   36.44 15.66 33.42
   

   View Slide

10. Some examples (CNN data)
    10
    Original text Transformed
    text
    James Best, best known for his portrayal of bumbling
    sheriff Rosco P. Coltrane on TV's "The Dukes of Hazzard,"
    died Monday after a brief illness. He was 88. Best died in
    hospice in Hickory, North Carolina, of complications from
    pneumonia, said Steve Latshaw, a longtime friend and
    Hollywood colleague.
    influential political
    minister dies in
    death
    

    View Slide

11. Model Limitations
    ● Currently not working very well with long texts
    ● Very slow in training
    ● Cannot understand out of vocabulary words
    11
    

    View Slide

12. What next?
    ● Hyperparameter tuning
    ● Use of Luong attention
    ● Reducing tokens
    12
    

    View Slide

13. Hyperparameter Tuning
    ● A hyperparameter is a parameter whose value is used to control the
    learning process.
    ● E.g. learning rate, number of hidden layer neurons, number of epochs
    ● Hyperparameter tuning is the problem of choosing a set of optimal
    hyperparameters for a learning algorithm.
    13
    

    View Slide

14. Drawback of Bahdanau Attention
    ● A major problem with Bahdanau attention model is that it tries to
    predict attention values for all hidden states.
    ● This needs a lot of computations.
    ● This is known as Global Attention.
    14
    

    View Slide

15. Luong Attention
    ● Luong attention solves these problems using the concept of Local
    attention.
    ● It tries to predict attention around a window of hidden states.
    ● Less computations mean less time for training.
    15
    

    View Slide

16. Using Pointer Generator Network (PGN)
    ● Current model is very heavy.
    ● Generates tokens for out of vocabulary words.
    ● Copying some part of input text can be helpful to generate summary.
    ● It will reduce training time as well as tokens.
    ● PGN can be said as a mixture of extractive and abstractive model.
    16
    

    View Slide

17. Original Text
    a day after winning nigeria’s presidency, muhammadu buhari told cnn’s christiane amanpour that
    he plans to aggressively fight corruption that has long plagued nigeria and go after the root of the
    nation’s unrest. buhari said he’ll “rapidly give attention” to curbing violence in the northeast part
    of nigeria, where the terrorist group boko haram operates. by cooperating with neighboring
    nations chad, cameroon and niger, he said his administration is confident it will be able to thwart
    criminals and others contributing to nigeria’s instability. for the first time in nigeria’s history, the
    opposition defeated the ruling party in democratic elections. buhari defeated incumbent
    goodluck jonathan by about 2 million votes, according to nigeria’s independent national electoral
    commission. the win comes after a long history of military rule, coups and botched attempts at
    democracy in africa’s most populous nation.
    17
    

    View Slide

18. With Attention
    UNK UNK says his administration is confident it will be able to destabilize nigeria’s economy.
    UNK says his administration is confident it will be able to thwart criminals and other nigerians. he
    says the country has long nigeria and nigeria’s economy.
    18
    

    View Slide

19. With PGN
    muhammadu buhari says he plans to aggressively fight corruption in the northeast part of nigeria.
    he says he’ll “rapidly give attention” to curbing violence in the northeast part of nigeria. he says
    his administration is confident it will be able to thwart criminals
    19
    

    View Slide

20. References
    ● https://arxiv.org/pdf/1704.04368.pdf
    ● https://arxiv.org/pdf/1509.00685.pdf
    ● https://arxiv.org/pdf/1508.04025.pdf
    ● https://www.coursera.org/learn/language-processing/lecture/RhxPO/get-to-the-point-summariz
    ation-with-pointer-generator-networks
    ● https://machinelearningmastery.com/global-attention-for-encoder-decoder-recurrent-neural-net
    works/
    ● http://cnyah.com/2017/08/01/attention-variants/
    20
    

    View Slide

21. Thank you!
    21
    

    View Slide