Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Text summarization Phase 2 Evaluation 2

Text summarization Phase 2 Evaluation 2

Phase 2 evaluation 2 of text summarization final year project, under professor U.A. Deshpande in collaboration with TCS, presentation.

Implementation of Seq2Seq model for auto text summarization using attention.

Phase 2 evaluation 1 presentation: https://speakerdeck.com/gautamabhishek46/text-summarization-phase-2-evaluation-1

Abhishek Gautam
Atharva Parwatkar
Sharvil Nagarkar

Professor in-charge: U. A. Deshpande
TCS Mentor : Dr. Sagar Sunkle

Abhishek Gautam

April 01, 2019

More Decks by Abhishek Gautam

Other Decks in Education


  1. Recap • Last time we discussed about: ◦ Abstractive Summarisation

    ◦ Seq2Seq Neural Network Model ◦ Attention mechanism 2
  2. Model Overview • Encoder-Decoder architecture ◦ Bidirectional LSTM as encoder

    ◦ Beam Search Decoder as decoder • Use of Bahdanau attention model 4
  3. Some examples 7 S. No. Original text Transformed text 1

    two more vietnamese professional football coaches have been suspended on suspicion of corruption or match-fixing in the country and abroad , officials said thursday . two more vietnamese football coaches suspended 2 the dollar inched higher against the euro thursday in thin trading on expectations the us economy saw robust job creation last month . dollar rises against euro 3 tourists from britain and ireland are flocking in ever greater numbers to the french riviera , helping make the region the second-most visited in france after paris , the area 's chamber of commerce said monday . < unk > tourists flock to france
  4. Model Evaluation • For evaluation we are using ROUGE scores.

    • We have used ROUGE-1, ROUGE-2 and ROUGE-L metrics • ROUGE-1: Counts 1-grams • ROUGE-2: Counts 2-grams • ROUGE-L: Uses Longest Common Subsequence(LCS) 8
  5. Evaluation Results 9 ROUGE-1 ROUGE-2 ROUGE-L Our Model 34.72 13.26

    31.49 Pointer Generator Network 36.44 15.66 33.42
  6. Some examples (CNN data) 10 Original text Transformed text James

    Best, best known for his portrayal of bumbling sheriff Rosco P. Coltrane on TV's "The Dukes of Hazzard," died Monday after a brief illness. He was 88. Best died in hospice in Hickory, North Carolina, of complications from pneumonia, said Steve Latshaw, a longtime friend and Hollywood colleague. influential political minister dies in death
  7. Model Limitations • Currently not working very well with long

    texts • Very slow in training • Cannot understand out of vocabulary words 11
  8. Hyperparameter Tuning • A hyperparameter is a parameter whose value

    is used to control the learning process. • E.g. learning rate, number of hidden layer neurons, number of epochs • Hyperparameter tuning is the problem of choosing a set of optimal hyperparameters for a learning algorithm. 13
  9. Drawback of Bahdanau Attention • A major problem with Bahdanau

    attention model is that it tries to predict attention values for all hidden states. • This needs a lot of computations. • This is known as Global Attention. 14
  10. Luong Attention • Luong attention solves these problems using the

    concept of Local attention. • It tries to predict attention around a window of hidden states. • Less computations mean less time for training. 15
  11. Using Pointer Generator Network (PGN) • Current model is very

    heavy. • Generates <unk> tokens for out of vocabulary words. • Copying some part of input text can be helpful to generate summary. • It will reduce training time as well as <unk> tokens. • PGN can be said as a mixture of extractive and abstractive model. 16
  12. Original Text a day after winning nigeria’s presidency, muhammadu buhari

    told cnn’s christiane amanpour that he plans to aggressively fight corruption that has long plagued nigeria and go after the root of the nation’s unrest. buhari said he’ll “rapidly give attention” to curbing violence in the northeast part of nigeria, where the terrorist group boko haram operates. by cooperating with neighboring nations chad, cameroon and niger, he said his administration is confident it will be able to thwart criminals and others contributing to nigeria’s instability. for the first time in nigeria’s history, the opposition defeated the ruling party in democratic elections. buhari defeated incumbent goodluck jonathan by about 2 million votes, according to nigeria’s independent national electoral commission. the win comes after a long history of military rule, coups and botched attempts at democracy in africa’s most populous nation. 17
  13. With Attention UNK UNK says his administration is confident it

    will be able to destabilize nigeria’s economy. UNK says his administration is confident it will be able to thwart criminals and other nigerians. he says the country has long nigeria and nigeria’s economy. 18
  14. With PGN muhammadu buhari says he plans to aggressively fight

    corruption in the northeast part of nigeria. he says he’ll “rapidly give attention” to curbing violence in the northeast part of nigeria. he says his administration is confident it will be able to thwart criminals 19