Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Not only Retrieval, But also Generation

Not only Retrieval, But also Generation

Scatter Lab Inc.

June 12, 2019

Transcript

39 slides
  1. !3 "An Adversarial Approach to High-Quality, Sentiment-Controlled Neural Dialogue Generation.”

    Kong, Xiang, et al. arXiv preprint arXiv:1901.07129 (2019). য়ט ࣗѐೡ ֤ޙ #Dialog-Generation #DialogSystem #CVAE #CGAN
  2. !4 ಽ۰Ҋ ೞח ޙઁ ࢎਊ੗੄ ؀ച History (੉੹ N-Turn) ী

    ؀ೠ ੸੺ೠ ׹߸ਸ Generation ೞ੗! ױ! х੿ਸ ઑ੺೧ࢲ ׹߸ਸ Generation ೡ ࣻ ੓ਸө? ৘: ݍ੓঻য? -> (ӛ੿) ਽! ૓૞ ݍ੓঻য / (ࠗ੿) ইפ ૓૞ ߹۽৓য
  3. !5 ಽ۰Ҋ ೞח ޙઁ ࢎਊ੗੄ ؀ച History (੉੹ N-Turn) ী

    ؀ೠ ੸੺ೠ ׹߸ਸ Generation ೞ੗! ױ! х੿ਸ ઑ੺೧ࢲ ׹߸ਸ Generation ೡ ࣻ ੓ਸө? ৘: ݍ੓঻য? -> (ӛ੿) ਽! ૓૞ ݍ੓঻য / (ࠗ੿) ইפ ૓૞ ߹۽৓য Q. যڌѱ ૓૞ ࢎۈэ਷ Quality ੄ ׹߸ਸ ٜ݅য յ Ԅؘ?
  4. !6 ಽ۰Ҋ ೞח ޙઁ ࢎਊ੗੄ ؀ച History (੉੹ N-Turn) ী

    ؀ೠ ੸੺ೠ ׹߸ਸ Generation ೞ੗! ױ! х੿ਸ ઑ੺೧ࢲ ׹߸ਸ Generation ೡ ࣻ ੓ਸө? ৘: ݍ੓঻য? -> (ӛ੿) ਽! ૓૞ ݍ੓঻য / (ࠗ੿) ইפ ૓૞ ߹۽৓য Q. যڌѱ ૓૞ ࢎۈэ਷ Quality ੄ ׹߸ਸ ٜ݅য յ Ԅؘ? Q. যڌѱ ౠ੿ х੿ী ؀ೠ ׹߸ਸ ٜ݅য յ Ԅؘ?
  5. !8 Motivation Sentimentח Dialog Systemী ੓যࢲ ࢎۈٜ੄ Engaigingҗ Interest੄ ࢚थਸ

    ݅ٞ ੉ী ؀ೠ ৈ۞ दبٜ੉ ੓঻૑݅, ؀ࠗ࠙੉ rule-based ژח template ӝ߈੄ ߑधਸ ࢎਊೣ → ੉ח જ਷ ׹߸ਸ ղӝ ਤ೧ࢲח ݆਷ మ೒݁җ ܙٜ੉ ೙ਃ۽ ೞח ޙઁ੼ End-to-End ӝ߈੄ Generation ߑߨٜ੉ Dialog System োҳীࢲ ઱ਃೠ ઱ઁ۽ ځয়ܰҊ ੓਺ → Reason: Rule-Base ࠁ׮ ഻न ਬোೞҊ ࢎਊೞӝ ਊ੉ೠ ߑߨী োҳ ૓೯
  6. !9 Previous Research ୡӝ োҳীࢲח SEQ2SEQ ӝ߈੄ ݽ؛ٜ੉ ݆੉ ࢎਊ

    غ঻ਵա, ੉റ ؊ જ਷ ௬ܻ౭৬ ੄بೠ ׹߸ਸ յ ࣻ ੓ب۾ ݽ؛੄ ࢿמ ೱ࢚ਸ ਤೠ ৈ۞ োҳٜ੉ ੉ܖয ઎਺. ؘ੉ఠܳ ӝ߈ਵ۽ Dialog End-to-End Modelীࢲ Sentiment ੿ࠁܳ Replyী ߈৔दః۰ח ݻݻ োҳٜ ৉द ઓ੤೮חؘ, ৈ۞ ޙઁ੼ٜ੉ ز߈೧ࢲ ߊࢤ ೮঻਺ 1.Hasegawa(2013): п emotion݃׮ ׮ܲ ݽ؛ਸ ೟णदఇ → ষ୒դ training-cost, sparsity 2. Shen et al (2017): latent variableী emotion infoܳ ֍঻૑݅, response੄ х੿ਸ যڌѱ ஶ౟܀ೡ૑ x 3.Zhou and Wang (2018): twitterী ੓ח emojiо ೧׼ contextܳ ߈৔ೡ Ѫ੉ۄ оࢸਸ ࣁ਋Ҋ ݽ؛ ࢸ҅, Cond Variational AutoEncoderਸ ੉ਊ೧ࢲ Sentimentܳ Control ೡ ࣻ ੓ח dialog-system ѐߊ
  7. !10 Previous Research 1. Hasegawa(2013): п emotion݃׮ ׮ܲ ݽ؛ਸ ೟णदఇ

    → ষ୒դ training-cost, sparsity 2. Shen et al (2017): latent variableী emotion infoܳ ֍঻૑݅, response੄ х੿ਸ যڌѱ ஶ౟܀ೡ૑ x 3. Zhou and Wang (2018): twitterী ੓ח emojiо ೧׼ contextܳ ߈৔ೡ Ѫ੉ۄ оࢸਸ ࣁ਋Ҋ ݽ؛ ࢸ҅, 
 Conditional Variational AutoEncoderਸ ੉ਊ೧ࢲ Sentimentܳ Control ೡ ࣻ ੓ח dialog-system ѐߊ
  8. !11 Proposal ੉ ֤ޙীࢲח Conditional Generative Adversarial Network (CGAN)җ 


    Conditional Variational Auto Encoder (CVAE) ӝ߈ਵ۽ х੿ਸ ઑ੺ೡ ࣻ ੓ח ؀ച ࢤࢿ ݽ؛ਸ ݅ٞ Fluency৬ Controlable فܻ݃ షՙܳ ׮ ੟ӝ ਤ೧ࢲ فѐ੄ ஹನք౟۽ ա־যઉ ੓਺: Generator, Discriminator Generatorח History৬ Emotion Labelਸ ੉ਊ೧ࢲ ׹߸ਸ Generation ೞҊ Discriminatorח History, Emotion Label, Responseܳ ࠁҊ ੉ѱ Ground Truthੋ૑ ݏ୶ѱ ೟ण(௏դ) Generatorо Discriminatorܳ ҅ࣘ ࣘ੉ب۾ ೟णೞݶࢲ ੗োझۣѱ Response ௬ܻ౭ܳ ֫੐
  9. VAE: Variational Auto Encoders Input X Reconstructed X Input Xܳ

    ౠ੿ Vector(Z: latent variable)۽ Abstraction दఃҊ ੉ܳ ׮द Reconstruct Encodeীࢲ աৡ Representationী ؀ೠ ࠁ׮ Ө਷ ੉೧
  10. Conditional GAN | VAE GANҗ VAE۽ ࢤࢿػ Ѿҗܳ ਋ܻо Control

    ೡ ࣻ হ਺ GANҗ VAE੄ ઁੌ ௾ ޙઁ੼ ৘: ࢎۈ੄ ঴ҷ Input ->Attribute(Ѩ੿࢝) ॆӖۄझܳ ՛ ࢎۈਸ ୹۱೧઻
  11. Conditional GAN | VAE Ӓ۞ݶ GAN, VAEܳ ೟णदఆ ٸ ౠ૚ب

    э੉ ֍ਵݶ غ૑ ঋਸө? Conditional GAN, VAE੄ Motivation
  12. Conditional GAN | VAE https://arxiv.org/abs/1702.01983 Antipov, Grigory, Moez Baccouche, and

    Jean-Luc Dugelay. "Face aging with conditional generative adversarial networks." 2017 IEEE International Conference on Image Processing (ICIP). IEEE, 2017. "Face aging with conditional generative adversarial networks."
  13. Conditional GAN | VAE https://arxiv.org/abs/1702.01983 Antipov, Grigory, Moez Baccouche, and

    Jean-Luc Dugelay. "Face aging with conditional generative adversarial networks." 2017 IEEE International Conference on Image Processing (ICIP). IEEE, 2017.
  14. Conditional Variational Auto Encoder 1. Utterance History Encoding -> Context

    Vec C 2. Embedding Sentiment Label -> Sentiment Vec S 3. Concat Context + Sentiment Vector = Z 4. Decoding Reply Sequence with Z
  15. ળࢿ: ղо যઁ ޤݡ঻ח૑ ঌই? ਌੤: উҾӘ೧ ળࢿ: ইפ ೠߣ

    ݏ୾ࠊ! Wh Dialog History Context Vec C + S = Z Latent Variable Conditional Variational Auto Encoder Encoder Wr ਌੤ : ঌѷয ޤ ݡ঻חؘ? Generator
  16. ળࢿ: ղо যઁ ޤݡ঻ח૑ ঌই? ਌੤: উҾӘ೧ ળࢿ: ইפ ೠߣ

    ݏ୾ࠊ! Wh Dialog History Context Vec C + S = Z Latent Variable Conditional Variational Auto Encoder Encoder Wr ਌੤ : ইפ উҾӘೞ׮Ҋ! Generator
  17. Conditional Variational Auto Encoder https://hugrypiggykim.com/2018/09/07/variational-autoencoder%EC%99%80-elboevidence-lower-bound/ ׹߸ Input -> Latent Variable(Z),

    х੿ чਸ ݏ୾ۄ! Latent Variable(Z) -> ׹߸, х੿ чਸ ݏ୾ۄ! ੉ فѐ੄ Objectܳ زदী ೟णೞח ߑध
  18. Conditional Variational Auto Encoder https://hugrypiggykim.com/2018/09/07/variational-autoencoder%EC%99%80-elboevidence-lower-bound/ ਌੤ : ইפ উҾӘೞ׮Ҋ! Wr

    х੿ : Context Vector(Z) : ળࢿ: ղо যઁ ޤݡ঻ח૑ ঌই? ਌੤: উҾӘ೧ ળࢿ: ইפ ೠߣ ݏ୾ࠊ! х੿ : Context Vector(Z) : ળࢿ: ղо যઁ ޤݡ঻ח૑ ঌই? ਌੤: উҾӘ೧ ળࢿ: ইפ ೠߣ ݏ୾ࠊ! ਌੤ : ইפ উҾӘೞ׮Ҋ! Wr
  19. Conditional Generative Adversarial Network 1. Utterance History Encoding -> Context

    Vec C 2. Embedding Sentiment Label -> Sentiment Vec S 3. Concat Context + Sentiment Vector = Z 4. Decoding Reply Sequence with Z 5. Discriminator(Decoded_output, Context, Sentiment)
  20. ળࢿ: ղо যઁ ޤݡ঻ח૑ ঌই? ਌੤: উҾӘ೧ ળࢿ: ইפ ೠߣ

    ݏ୾ࠊ! Wh Dialog History Context Vec C + S = Z Latent Variable Encoder Wr ਌੤ : ঌѷয ޤ ݡ঻חؘ? Generator Conditional Generative Adversarial Network
  21. ળࢿ: ղо যઁ ޤݡ঻ח૑ ঌই? ਌੤: উҾӘ೧ ળࢿ: ইפ ೠߣ

    ݏ୾ࠊ! Wh Dialog History Context Vec C + S = Z Latent Variable Encoder Wr ਌੤ : ঌѷয ޤ ݡ঻חؘ? Generator Discriminator ੉Ѥ ૓૞ ਌੤о ೠ ݈ੌө ইפݶ Generatorо ࢤࢿೠ ݈ੌө? Conditional Generative Adversarial Network
  22. ળࢿ: ղо যઁ ޤݡ঻ח૑ ঌই? ਌੤: উҾӘ೧ ળࢿ: ইפ ೠߣ

    ݏ୾ࠊ! Wh Dialog History Context Vec C + S = Z Latent Variable Encoder Wr ਌੤ : ঌѷয ޤ ݡ঻חؘ? Generator Discriminator ੉Ѥ ૓૞ ਌੤о ೠ ݈ੌө ইפݶ Generatorо ࢤࢿೠ ݈ੌө? Discriminatorܳ ࣘ੉ӝ ਤ೧ࢲ ؊ ਌੤୊ۢ ੉ঠӝ ೡకঠ Conditional Generative Adversarial Network
  23. ࢿמ: ੿۝ಣо ױࣽೠ SEQ2SEQ ਵ۽ח ੿݈ ࢿמ੉ ߹۽ જ૑ ঋ਺

    (ACC 55%, PPL 150+) ੹୓੸ਵ۽ CVAEܳ ా೧ࢲ ೟णػ Generation੉ જ਷ ࢿמਸ ࠁ੉ח Ѫ э਺ (ACC 75%, PPL 81) CGAN݅ਸ ੉ਊ೧ࢲח ௼ѱ જਸ ࢿמਸ ࠁ૑ ޅೣ -> ࢎۈਸ ൑ղղח Ѫ ݅ਵ۽ח ൨ٞ (ACC 64%, PPL 120)
  24. ࢿמ: ੿۝ಣо CVAEо ೧֬਷ ߏী CGAN ઑ޷ܐ ஖פ SOTA Generation

    ػ Replyী ؀ೠ ࠁ׮ Ө਷ ੉೧ : CVAE ࠁ׮ ࢎۈ୊ۢ ׹߸ೡ ࣻ ੓ח ൑ղղӝ : CGAN
  25. ୨ಣ Context-Aware Pretrained BERT۽ ؀ച ࢤࢿ పझ௼ܳ Transfer-Learning ೞݶࢲ 


    ࢤࢿػ ׹߸ੋ૑ ইפݶ ૓૞ पઁ ؀ചੋ૑ ҳ࠙ೞח Discriminatorܳ ୶о೧ࢲ ੗োझۣѱ ؀ച ௬ܻ౭ܳ ֫ৈ р׮ݶ ૓૞ ࢎۈ э਷ ؀ചܳ ࢤࢿೞח ࠈਸ ٜ݅ ࣻ ੓૑ ঋਸө?!?! → Conditionalਸ ୶оೞݶ Blackbox rate ઴ੌ ࣻ ੓਺ ޥо ૓૞ ೧ࠁҊ र׮!! ࢎۈ୊ۢ ҳয୓۽ ؀׹ೡ ࣻ ੓ਸө??

More Decks by Scatter Lab Inc.

Other Decks in Research