Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Deep Collaborative Learning by Knowledge Transfer Graph

Deep Collaborative Learning by Knowledge Transfer Graph

In this talk I will present our latest research about knowledge transfer graph for Deep Collaborative Learning (DCL), which is a method that incorporates Knowledge Distillation and Deep Mutual Learning. DCL is represented by a directional graph where each model is represented by a node, and the propagation of knowledge from the source node to the target node is represented by edges. In DCL, a hyperparameter search can be used to search for an optimal knowledge transfer graph.

Hironobu Fujiyoshi

August 12, 2022
Tweet

More Decks by Hironobu Fujiyoshi

Other Decks in Research

Transcript

  1. %FFQ$PMMBCPSBUJWF-FBSOJOHCZ,OPXMFEHF5SBOTGFS(SBQI
    )JSPOPCV'VKJZPTIJ $IVCV6OJWFSTJUZ

    IUUQNQSHKQ

    View full-size slide

  2. #JPHSBQIZ%S)JSPOPCV'VKJZPTIJ
    2
    த෦େֶϩΰ
    த෦େֶϩΰ

    w ɿ1I%GSPN$IVCV6OJWFSTJUZ
    w ɿ1PTUEPDUPSBMGFMMPX!$BSOFHJF.FMMPO6OJWFSTJUZ
    r 7JEFP4VSWFJMMBODFBOE.POJUPSJOH 74".
    )VNBOPJE7JTJPO
    w QSFTFOUɿ$IVCV6OJWFSTJUZ
    w ɿ7JTJUJOHSFTFBSDIFS!$BSOFHJF.FMMPO6OJWFSTJUZ
    r 1FPQMF*NBHF"OBMZTJT
    w QSFTFOUɿ1SPGFTTPS $IVCV6OJWFSTJUZ
    7JEFP4VSWFJMMBODFBOE.POJUPSJOH<1SPDPG*&&&`> lTUBSzTLFMUPOJ[BUJPO<8"$7`> )VNBOPJEWJTJPOQSPKFDU $.6)POEB

    View full-size slide

  3. %FFQDPMMBCPSBUJWFMFBSOJOHCZLOPXMFEHFUSBOTGFSHSBQI<"$$7>
    த෦େֶϩΰ
    த෦େֶϩΰ

    View full-size slide

  4. (PJOHEFFQFSUPJNQSPWFQFSGPSNBODF
    த෦େֶϩΰ
    த෦େֶϩΰ
    w %FFQFSOFUXPSLXPOBUMBUFTUDPNQFUJUJPO
    2012
    SuperVision GoogLeNet
    Konvolüzasyon
    Pooling
    Softmax
    Diğer
    [Krizhevsky NIPS 2012] [Szegedy arxiv 2014]-22 [Sim
    "MFY/FU MBZFST
    *-473$

    2014
    GoogLeNet
    Konvolüzasyon
    Pooling
    Softmax
    Diğer
    VGG MSRA
    [Szegedy arxiv 2014]-22 [Simonyan arxiv 2014] -19 [He arxiv 2014]
    n
    2014
    GoogLeNet
    Konvolüzasyon
    Pooling
    Softmax
    Diğer
    VGG MSRA
    35/36 t
    derin öğ
    kullanm
    20/36 t
    açık-kay
    Caffe
    uygula
    kullanm
    012] [Szegedy arxiv 2014]-22 [Simonyan arxiv 2014] -19 [He arxiv 2014]
    7(( MBZFST
    *-473$

    (PPHMF/FU MBZFST
    *-473$

    14
    3FT/FU MBZFST
    *-473$


    ˠNJMMJPO
    ˠNJMMJPO
    ˠNJMMJPO

    View full-size slide

  5. ,OPXMFEHF%JTUJMMBUJPO ,%
    <()JOUPO >
    த෦େֶϩΰ
    த෦େֶϩΰ
    w ,OPXMFEHFUSBOTGFSGSPNUFBDIFSOFUXPSLUPTUVEFOUOFUXPSL
    5FBDIFSQSFUSBJOFEMBSHFOFUXPSL
    4UVEFOUTNBMMOFUXPSL

    5FBDIFS
    /FUXPSL
    4UVEFOU
    /FUXPSL
    -BSHF

    QSFUSBJOFE
    4NBMM
    ,OPXMFEHFUSBOTGFS
    ,OPXMFEHF
    %JTUJMMBUJPO
    ,OPXMFEHF

    View full-size slide

  6. w 5FBDIFSˠ4UVEFOU
    5FBDIFSQSFUSBJOFEMBSHFOFUXPSL
    4UVEFOUTNBMMOFUXPSL
    5SBJOJOHTUVEFOUOFUXPSLJTUSBJOFECBTFEPOIBSEUBSHFUBOETPGUUBSHFU
    ,OPXMFEHF%JTUJMMBUJPO ,%
    <()JOUPO >
    த෦େֶϩΰ
    த෦େֶϩΰ

    %BSL,OPXMFEHF
    5FBDIFS
    4UVEFOU
    $SPTT&OUSPQZ
    $SPTT&OUSPQZ
    MBCFM
    QSFUSBJOFE

    4PGUUBSHFU
    )BSEUBSHFU
    ʢDPSSFDUMBCFMʣ
    p1
    p2
    ʢQSPCBCJMJUZEJTUSJCVUJPOʣ
    #BDLQSPQ

    View full-size slide

  7. %FFQ.VUVBM-FBSOJOH %.-
    <:;IBOH >
    த෦େֶϩΰ
    த෦େֶϩΰ
    w 4UVEFOU⁶4UVEFOU
    "OFOTFNCMFPGTUVEFOUTMFBSODPMMBCPSBUJWFMZBOEUFBDIFBDIPUIFSUISPVHIPVUUIF
    USBJOJOHQSPDFTT
    ,VMMCBDL-FJCMFS ,-
    %JWFSHFODFJTVTFEUPNBUDIUIFUXPOFUXPSLQSFEJDUJPOT BOE
    p1
    p2

    MBCFM
    4UVEFOU
    4UVEFOU
    p2
    KL(p1
    ||p2
    )
    KL(p2
    ||p1
    )
    ˠ%.-NBLFTUIFTUVEFOUOFUXPSLCFUUFSUIBOUIFPOFVTFEJO,%
    )BSE5BSHFU
    $SPTT&OUSPQZ

    4PGU5BSHFU
    ,-EJWFSHFODF

    p1

    View full-size slide

  8. %FFQ.VUVBM-FBSOJOH %.-
    <:;IBOH >
    த෦େֶϩΰ
    த෦େֶϩΰ
    w 8IZ%.-XPSLTCFUUFS

    *OEFQFOEFOU
    Y Zd Y Zd
    %.-
    7*46"-*;*/(5)&-044-"/%4$"1&0'/&63"-/&54
    ˠ%.-TFBSDIFTGPSQPJOUTXIFSFUIFMPTTMBOETDBQFJT
    fl
    BU

    View full-size slide

  9. %FFQDPMMBCPSBUJWFMFBSOJOH
    த෦େֶϩΰ
    த෦େֶϩΰ
    w ,OPXMFEHFUSBOTGFSCFUXFFONVMUJQMFOFUXPSLT
    ,OPXMFEHF%JTUJMMBUJPO ,%
    <()JOUPO >
    %FFQ.VUVBM-FBSOJOH %.-
    <:;IBOH >

    5FBDIFS
    4UVEFOU
    ,OPXMFEHF%JTUJMMBUJPO ,%

    4UVEFOU
    4UVEFOU
    %FFQ.VUVBM-FBSOJOH %.-

    p1
    p2
    p1
    p2

    View full-size slide

  10. 7BSJBUJPOTPG,%BOE%.-
    த෦େֶϩΰ
    த෦େֶϩΰ

    4UVEFOU
    4UVEFOU
    4UVEFOU
    4UVEFOU
    4UVEFOU
    4UVEFOU
    4UVEFOU
    5FBDIFS
    4UVEFOU
    5FBDIFS
    4UVEFOU
    5FBDIFS
    5"
    4UVEFOU
    ,OPXMFEHF%JTUJMMBUJPO
    %FFQ.VUVBM-FBSOJOH
    ,OPXMFEHF%JTUJMMBUJPO
    <()JOUPO >
    #PSO"HBJO
    <'VSMBOFMMP >
    5FBDIFS"TTJTUBOU
    <.JS[BEFI >
    %FFQ.VUVBM-FBSOJOH<:;IBOH >
    -BSHF4NBMM 4BNF 4UFQXJTF
    -BSHF4NBMM 4BNF PGNPEFM
    p1
    p2
    p1
    p2
    p1
    p2
    p1
    p2
    p1
    p2
    p3
    p1
    p2
    p3

    View full-size slide

  11. த෦େֶϩΰ
    த෦େֶϩΰ

    ,OPXMFEHF%JTUJMMBUJPO
    <()JOUPO >
    #PSO"HBJO
    <'VSMBOFMMP >
    5FBDIFS"TTJTUBOU
    <.JS[BEFI >
    %FFQ.VUVBM-FBSOJOH<:;IBOH >
    ,OPXMFEHF%JTUJMMBUJPO
    %FFQ.VUVBM-FBSOJOH
    5IFTFBQQSPBDIFTPGEFFQDPMMBCPSBUJWFMFBSOJOHBSFEFTJHOFECZSFTFBSDIFST IVNBO

    7BSJBUJPOTPG,%BOE%.-
    -BSHF4NBMM 4BNF 4UFQXJTF
    -BSHF4NBMM 4BNF PGNPEFM

    View full-size slide

  12. 0VSHPBM
    த෦େֶϩΰ
    த෦େֶϩΰ

    w 4DBMFVQEFFQDPMMBCPSBUJWFMFBSOJOHUPDMBTTSPPNTJ[F
    %FFQBOEEJWFSTFDPMMBCPSBUJWFMFBSOJOH
    ˠ%FFQDPMMBCPSBUJWFMFBSOJOHTJNJMBSUPSFBMDMBTTSPPNUIBUIBWFTUVEFOUT
    XJUIEJ
    ff
    FSFOUQFSTPOBMJUJFT

    View full-size slide

  13. w 4DBMFVQEFFQDPMMBCPSBUJWFMFBSOJOHUPDMBTTSPPNTJ[F
    %FFQBOEEJWFSTFDPMMBCPSBUJWFMFBSOJOH
    w ,OPXMFEHF5SBOTGFS(SBQI<.JOBNJ "$$7>
    "OPWFMHSBQISFQSFTFOUBUJPOPGLOPXMFEHFUSBOTGFSGPSEFFQDPMMBCPSBUJWFMFBSOJOH
    த෦େֶϩΰ
    த෦େֶϩΰ

    0VSHPBM
    ,% %.-
    ,OPXMFEHFUSBOTGFSHSBQI
    𝑚
    3
    𝑚
    1
    𝑚
    2
    𝐿
    ^
    𝑦
    ,1
    𝐿
    ^
    𝑦
    ,2
    𝐿
    ^
    𝑦
    ,3
    𝐿
    1,2
    𝐿
    1,3
    𝐿
    2,1
    𝐿
    3,1
    𝐿
    3,2
    𝐿
    2,3
    ^
    𝑦
    ^
    𝑦
    ^
    𝑦

    View full-size slide

  14. w (SBQISFQSFTFOUBUJPO
    /PEF%FFQMFBSOJOHNPEFM
    &EHFɿ-PTTPGLOPXMFEHFUSBOTGFS
    (SBQISFQSFTFOUBUJPOPGDPOWFOUJPOBMNFUIPET
    த෦େֶϩΰ
    த෦େֶϩΰ

    ,OPXMFEHF%JTUJMMBUJPO
    ,%

    5FBDIFS
    4UVEFOU
    4UVEFOU
    4UVEFOU
    %FFQ.VUVBM-FBSOJOH
    %.-

    p1
    p2
    p1
    p2
    -BSHF 4NBMM
    m1
    m2
    POFEJSFDUJPOBMFEHF

    -BSHF 4NBMM
    m1
    m2
    CJEJSFDUJPOBMFEHFT

    (SBQISFQSFTFOUBUJPO

    View full-size slide

  15. w "VYJMJBSZOPEFTTVQQPSUUIFUSBJOJOHPGUIFUBSHFUOPEF
    /PEF%FFQMFBSOJOHNPEFM
    &EHFɿ-PTTPGLOPXMFEHFUSBOTGFS
    ,OPXMFEHFUSBOTGFSHSBQI OPEFT

    த෦େֶϩΰ
    த෦େֶϩΰ

    𝑚
    3
    𝑚
    1
    𝑚
    2
    𝐿
    ^
    𝑦
    ,1
    𝐿
    ^
    𝑦
    ,2
    𝐿
    ^
    𝑦
    ,3
    𝐿
    1,2
    𝐿
    1,3
    𝐿
    2,1
    𝐿
    3,1
    𝐿
    3,2
    𝐿
    2,3
    ^
    𝑦
    ^
    𝑦
    ^
    𝑦
    DMBTTMBCFM
    5BSHFUOPEF
    3FT/FU
    "VYJMJBSZOPEFT
    3FT/FU
    8JEF3FT/FU
    %FOTF/FU

    View full-size slide

  16. w &BDIFEHFJTEF
    fi
    OFECZBEJ
    ff
    FSFOUUZQFPGMPTTGVODUJPO
    5IJTFOBCMFTVTUPNBLFBEJWFSTFLOPXMFEHFUSBOTGFS
    த෦େֶϩΰ
    த෦େֶϩΰ

    𝑚
    3
    𝑚
    1
    𝑚
    2
    𝐿
    ^
    𝑦
    ,1
    𝐿
    ^
    𝑦
    ,2
    𝐿
    ^
    𝑦
    ,3
    𝐿
    1,2
    𝐿
    1,3
    𝐿
    2,1
    𝐿
    3,1
    𝐿
    3,2
    𝐿
    2,3
    ^
    𝑦
    ^
    𝑦
    ^
    𝑦
    -PTTGVODUJPO
    𝐿
    =
    𝐻
    (
    𝑝
    ^
    𝑦
    ,
    𝑝
    𝑛
    )
    𝐿
    =
    𝐾
    𝐿
    (
    𝑝
    𝑛
    ||
    𝑝
    𝑚
    )
    𝐿
    = 0

    ,OPXMFEHFUSBOTGFSHSBQI OPEFT

    View full-size slide

  17. w ,OPXMFEHF%JTUJMMBUJPO ,%

    0OFEJSFDUFELOPXMFEHFUSBOTGFS
    த෦େֶϩΰ
    த෦େֶϩΰ

    ,%JOLOPXMFEHFUSBOTGFSHSBQI
    𝑚
    2
    𝐿
    ^
    𝑦
    ,1
    𝐿
    ^
    𝑦
    ,2
    𝐿
    ^
    𝑦
    ,3
    𝐿
    1,2
    𝐿
    1,3
    𝐿
    2,1
    𝐿
    3,2
    𝐿
    2,3
    ^
    𝑦
    ^
    𝑦
    ^
    𝑦
    𝑚
    3
    𝑚
    1
    5BSHFUOPEF
    𝐿
    3,1

    View full-size slide

  18. w %FFQ.VUVBM-FBSOJOH %.-

    #JEJSFDUFELOPXMFEHFUSBOTGFS
    𝐿
    ^
    𝑦
    ,3
    𝐿
    1,3
    𝐿
    3,2
    𝐿
    2,3
    ^
    𝑦
    𝑚
    3
    𝐿
    3,1
    த෦େֶϩΰ
    த෦େֶϩΰ

    %.-JOLOPXMFEHFUSBOTGFSHSBQI
    𝑚
    2
    𝐿
    ^
    𝑦
    ,1
    𝐿
    ^
    𝑦
    ,2
    𝐿
    1,2
    𝐿
    2,1
    ^
    𝑦
    ^
    𝑦
    𝑚
    1
    5BSHFUOPEF

    View full-size slide

  19. w "TBSFTVMUPGTFBSDIJOH UIFCFTUDPNCJOBUJPOPGMPTTGVODUJPOTDBOSFQSFTFOUB
    OPWFMEFFQDPMMBCPSBUJWFMFBSOJOH
    த෦େֶϩΰ
    த෦େֶϩΰ

    𝑚
    3
    𝑚
    1
    𝑚
    2
    𝐿
    ^
    𝑦
    ,1
    𝐿
    ^
    𝑦
    ,2
    𝐿
    ^
    𝑦
    ,3
    𝐿
    1,2
    𝐿
    1,3
    𝐿
    2,1
    𝐿
    3,1
    𝐿
    3,2
    𝐿
    2,3
    ^
    𝑦
    ^
    𝑦
    ^
    𝑦
    ,OPXMFEHFUSBOTGFSHSBQI OPEFT

    5BSHFUOPEF

    View full-size slide

  20. w ,OPXMFEHFUSBOTGFSGSPNOPEF TPVSDF
    UPOPEF EFTUJOBUJPO

    -PTTGVODUJPOPGLOPXMFEHFUSBOTGFSHSBQI
    த෦େֶϩΰ
    த෦େֶϩΰ

    𝑚
    3
    𝑚
    1
    𝑚
    2
    𝐿
    ^
    𝑦
    ,1
    𝐿
    ^
    𝑦
    ,2
    𝐿
    ^
    𝑦
    ,3
    𝐿
    1,2
    𝐿
    1,3
    𝐿
    2,1
    𝐿
    3,1
    𝐿
    3,2
    𝐿
    2,3
    ^
    𝑦
    ^
    𝑦
    ^
    𝑦
    𝑚
    1
    𝑚
    2
    𝐿
    2,1
    𝑚
    3
    𝑚
    1
    𝑚
    2
    𝐿
    ^
    𝑦
    ,1
    𝐿
    ^
    𝑦
    ,2
    𝐿
    ^
    𝑦
    ,3
    𝐿
    1,2
    𝐿
    1,3
    𝐿
    2,1
    𝐿
    3,1
    𝐿
    3,2
    𝐿
    2,3
    ^
    𝑦
    ^
    𝑦
    ^
    𝑦
    5BSHFUOPEF

    View full-size slide

  21. w ,OPXMFEHFUSBOTGFSGSPNOPEF TPVSDF
    UPOPEF EFTUJOBUJPO

    -PTTGVODUJPOPGLOPXMFEHFUSBOTGFSHSBQI
    த෦େֶϩΰ
    த෦େֶϩΰ

    𝑚
    3
    𝑚
    1
    𝑚
    2
    𝐿
    ^
    𝑦
    ,1
    𝐿
    ^
    𝑦
    ,2
    𝐿
    ^
    𝑦
    ,3
    𝐿
    1,2
    𝐿
    1,3
    𝐿
    2,1
    𝐿
    3,1
    𝐿
    3,2
    𝐿
    2,3
    ^
    𝑦
    ^
    𝑦
    ^
    𝑦
    𝑚
    1
    𝑚
    2
    𝐿
    2,1
    -PTT
    GVOD
    4PVSDF %FTUJOBUJPO
    𝑚
    2
    𝑚
    1
    p2
    (c|x) p1
    (c|x)
    L2,1
    (p2
    , p1
    )
    'PSXBSE

    View full-size slide

  22. w ,OPXMFEHFUSBOTGFSGSPNOPEF TPVSDF
    UPOPEF EFTUJOBUJPO

    -PTTGVODUJPOPGLOPXMFEHFUSBOTGFSHSBQI
    த෦େֶϩΰ
    த෦େֶϩΰ

    𝑚
    3
    𝑚
    1
    𝑚
    2
    𝐿
    ^
    𝑦
    ,1
    𝐿
    ^
    𝑦
    ,2
    𝐿
    ^
    𝑦
    ,3
    𝐿
    1,2
    𝐿
    1,3
    𝐿
    2,1
    𝐿
    3,1
    𝐿
    3,2
    𝐿
    2,3
    ^
    𝑦
    ^
    𝑦
    ^
    𝑦
    𝑚
    1
    𝑚
    2
    𝐿
    2,1
    -PTT
    GVOD
    4PVSDF %FTUJOBUJPO
    𝑚
    2
    𝑚
    1
    L2,1
    (p2
    , p1
    )
    #BDLQSPQ
    %FUBDI
    #BDLXBSE

    View full-size slide

  23. w ,OPXMFEHFUSBOTGFSGSPNOPEF TPVSDF
    UPOPEF EFTUJOBUJPO

    த෦େֶϩΰ
    த෦େֶϩΰ

    𝑚
    3
    𝑚
    1
    𝑚
    2
    𝐿
    ^
    𝑦
    ,1
    𝐿
    ^
    𝑦
    ,2
    𝐿
    ^
    𝑦
    ,3
    𝐿
    1,2
    𝐿
    1,3
    𝐿
    2,1
    𝐿
    3,1
    𝐿
    3,2
    𝐿
    2,3
    ^
    𝑦
    ^
    𝑦
    ^
    𝑦
    𝑚
    1
    𝑚
    2
    𝐿
    2,1
    -PTT
    GVOD
    4PVSDF %FTUJOBUJPO
    𝑚
    2
    𝑚
    1
    L2,1
    (p2
    , p1
    )
    Gate
    KL div
    'PSXBSE
    p2
    (c|x) p1
    (c|x)
    -PTTGVODUJPOPGLOPXMFEHFUSBOTGFSHSBQI

    View full-size slide

  24. w (BUFGVODUJPOTDPOUSPMIPXUIFLOPXMFEHFJTUSBOTGFSSFE
    (BUFGVODUJPOTGPSMPTTGVODUJPO
    த෦େֶϩΰ
    த෦େֶϩΰ

    𝑚
    3
    𝑚
    1
    𝑚
    2
    𝐿
    ^
    𝑦
    ,1
    𝐿
    ^
    𝑦
    ,2
    𝐿
    ^
    𝑦
    ,3
    𝐿
    1,2
    𝐿
    1,3
    𝐿
    2,1
    𝐿
    3,1
    𝐿
    3,2
    𝐿
    2,3
    ^
    𝑦
    ^
    𝑦
    ^
    𝑦
    𝑚
    1
    𝑚
    2
    𝐿
    2,1
    4PVSDF %FTUJOBUJPO
    𝑚
    2
    𝑚
    1
    L2,1
    (p2
    , p1
    )
    'PSXBSE
    p2
    (c|x) p1
    (c|x)
    Gate
    KL div
    $VUPGG(BUF
    -JOFBS(BUF
    5ISPVHI(BUF
    $PSSFDU(BUF

    View full-size slide

  25. w (BUFGVODUJPOTDPOUSPMIPXUIFLOPXMFEHFJTUSBOTGFSSFE
    5ISPVHI(BUF
    த෦େֶϩΰ
    த෦େֶϩΰ

    𝑚
    3
    𝑚
    1
    𝑚
    2
    𝐿
    ^
    𝑦
    ,1
    𝐿
    ^
    𝑦
    ,2
    𝐿
    ^
    𝑦
    ,3
    𝐿
    1,2
    𝐿
    1,3
    𝐿
    2,1
    𝐿
    3,1
    𝐿
    3,2
    𝐿
    2,3
    ^
    𝑦
    ^
    𝑦
    ^
    𝑦
    𝑚
    1
    𝑚
    2
    𝐿
    2,1
    4PVSDF %FTUJOBUJPO
    𝑚
    2
    𝑚
    1
    L2,1
    (p2
    , p1
    )
    $VUPGG(BUF
    -JOFBS(BUF
    $PSSFDU(BUF
    5ISPVHI(BUF
    𝐺
    (
    𝐷
    𝐾 𝐿
    ) =
    𝐷
    𝐾 𝐿
    /PDIBOHF
    QBTTJOHUISPVHI
    'PSXBSE
    p2
    (c|x) p1
    (c|x)
    Gate
    KL div

    View full-size slide

  26. w $VUP
    ff
    (BUFJTBHBUFUIBUBMXBZTPVUQVUT[FSP
    $VUP
    ff
    (BUF
    த෦େֶϩΰ
    த෦େֶϩΰ

    𝑚
    3
    𝑚
    1
    𝑚
    2
    𝐿
    ^
    𝑦
    ,1
    𝐿
    ^
    𝑦
    ,2
    𝐿
    ^
    𝑦
    ,3
    𝐿
    1,2
    𝐿
    1,3
    𝐿
    2,1
    𝐿
    3,1
    𝐿
    3,2
    𝐿
    2,3
    ^
    𝑦
    ^
    𝑦
    ^
    𝑦
    𝑚
    1
    𝑚
    2
    𝐿
    2,1
    4PVSDF %FTUJOBUJPO
    𝑚
    2
    𝑚
    1
    L2,1
    (p2
    , p1
    )
    $VUPGG(BUF
    -JOFBS(BUF
    $PSSFDU(BUF
    5ISPVHI(BUF
    0VUQVUT[FSP
    $VUUJOHUIFFEHF

    𝐺
    (
    𝐷
    𝐾 𝐿
    ) = 0
    'PSXBSE
    p2
    (c|x) p1
    (c|x)
    Gate
    KL div

    View full-size slide

  27. w -JOFBS(BUFJTBHBUFUIBUPVUQVUTHSBEVBMMZPWFSUJNF
    -JOFBS(BUF
    த෦େֶϩΰ
    த෦େֶϩΰ

    𝑚
    3
    𝑚
    1
    𝑚
    2
    𝐿
    ^
    𝑦
    ,1
    𝐿
    ^
    𝑦
    ,2
    𝐿
    ^
    𝑦
    ,3
    𝐿
    1,2
    𝐿
    1,3
    𝐿
    2,1
    𝐿
    3,1
    𝐿
    3,2
    𝐿
    2,3
    ^
    𝑦
    ^
    𝑦
    ^
    𝑦
    𝑚
    1
    𝑚
    2
    𝐿
    2,1
    4PVSDF %FTUJOBUJPO
    𝑚
    2
    𝑚
    1
    L2,1
    (p2
    , p1
    )
    $VUPGG(BUF
    -JOFBS(BUF
    $PSSFDU(BUF
    5ISPVHI(BUF
    Gate
    KL div
    0VUQVUTHSBEVBMMZ
    PWFSUJNF
    𝐺
    (
    𝐷
    𝐾
    𝐿
    ) =
    𝑡
    𝑡
    𝑚 𝑎
    𝑥

    𝐷
    𝐾 𝐿
    'PSXBSE
    p2
    (c|x) p1
    (c|x)

    View full-size slide

  28. w 1BTTFTUISPVHIUIFJOQVU XIFOUIFQSFEJDUJPOPGOPEF TPVSDF
    JTDPSSFDU
    $PSSFDUHBUF
    த෦େֶϩΰ
    த෦େֶϩΰ

    𝑚
    3
    𝑚
    1
    𝑚
    2
    𝐿
    ^
    𝑦
    ,1
    𝐿
    ^
    𝑦
    ,2
    𝐿
    ^
    𝑦
    ,3
    𝐿
    1,2
    𝐿
    1,3
    𝐿
    2,1
    𝐿
    3,1
    𝐿
    3,2
    𝐿
    2,3
    ^
    𝑦
    ^
    𝑦
    ^
    𝑦
    𝑚
    1
    𝑚
    2
    𝐿
    2,1
    4PVSDF %FTUJOBUJPO
    𝑚
    2
    𝑚
    1
    L2,1
    (p2
    , p1
    )
    $VUPGG(BUF
    -JOFBS(BUF
    $PSSFDU(BUF
    5ISPVHI(BUF
    Gate
    KL div
    0VUQVUTPOMZJGUIF
    QSFEJDUJPOJTDPSSFDU
    𝐺
    (
    𝐷
    𝐾
    𝐿
    ) =
    𝛿
    ^
    𝑦
    ,
    𝑦
    𝑚
    2

    𝐷
    𝐾 𝐿
    'PSXBSE
    p2
    (c|x) p1
    (c|x)

    View full-size slide

  29. w (BUFGVODUJPOTDPOUSPMIPXUIFLOPXMFEHFJTUSBOTGFSSFE
    (BUFGVODUJPOTGPSMPTTGVODUJPO
    த෦େֶϩΰ
    த෦େֶϩΰ

    𝑚
    3
    𝑚
    1
    𝑚
    2
    𝐿
    ^
    𝑦
    ,1
    𝐿
    ^
    𝑦
    ,2
    𝐿
    ^
    𝑦
    ,3
    𝐿
    1,2
    𝐿
    1,3
    𝐿
    2,1
    𝐿
    3,1
    𝐿
    3,2
    𝐿
    2,3
    ^
    𝑦
    ^
    𝑦
    ^
    𝑦
    𝑚
    1
    𝑚
    2
    𝐿
    2,1
    4PVSDF %FTUJOBUJPO
    𝑚
    2
    𝑚
    1
    L2,1
    (p2
    , p1
    )
    'PSXBSE
    p2
    (c|x) p1
    (c|x)
    Gate
    KL div
    $VUPGG(BUF
    -JOFBS(BUF
    5ISPVHI(BUF
    $PSSFDU(BUF

    View full-size slide

  30. w "QQMZIZQFSQBSBNFUFSTFBSDIUPPQUJNJ[FLOPXMFEHFUSBOTGFSHSBQI
    0QUJNJ[BUJPO"TZODISPOPVT4VDDFTTJWF)BMWJOH"MHPSJUIN "4)"

    1BSBNFUFST(BUFGVODUJPOT "VYJMJBSZOPEFT
    0QUJNJ[JOHLOPXMFEHFUSBOTGFSHSBQI
    த෦େֶϩΰ
    த෦େֶϩΰ

    𝑚
    3
    𝑚
    1
    𝑚
    2
    𝐿
    ^
    𝑦
    ,1
    𝐿
    ^
    𝑦
    ,2
    𝐿
    ^
    𝑦
    ,3
    𝐿
    1,2
    𝐿
    1,3
    𝐿
    2,1
    𝐿
    3,1
    𝐿
    3,2
    𝐿
    2,3
    ^
    𝑦
    ^
    𝑦
    ^
    𝑦
    (BUFGVODUJPO
    • 5ISPVHI(BUF
    • $VUPGG(BUF
    • -JOFBS(BUF
    • $PSSFDU(BUF
    • 3FT/FU
    5BSHFUOPEF
    • 3FT/FU
    • 3FT/FU
    • 8JEF3FT/FU
    "VYJMJBSZOPEF
    /PPGDPNCJOBUJPOT ʢOPEFTʣ

    View full-size slide

  31. 0QUJNJ[JOHLOPXMFEHFUSBOTGFSHSBQI
    த෦େֶϩΰ
    த෦େֶϩΰ

    /PPG(16TFSWFSTɿ
    4FBSDIJOH ɿUJNFT
    'SBNFXPSL ɿ0QUVOB

    View full-size slide

  32. w 5BSHFUOPEF 3FT/FU

    w 7BOJMMBNPEFM
    #FTULOPXMFEHFUSBOTGFSHSBQI OPEFT

    த෦େֶϩΰ
    த෦େֶϩΰ

    5BSHFUOPEF
    "VYJMJBSZOPEF
    QSFUSBJOFE

    "VYJMJBSZOPEF

    View full-size slide

  33. w 5BSHFUOPEF 3FT/FU

    w 7BOJMMBNPEFM
    #FTULOPXMFEHFUSBOTGFSHSBQI OPEFT

    த෦େֶϩΰ
    த෦େֶϩΰ

    "VYJMJBSZOPEF
    QSFUSBJOFE

    "VYJMJBSZOPEF
    5BSHFUOPEF
    ,OPXMFEHF
    %JTUJMMBUJPO
    5FBDIFSOFUXPSL
    'JSTU ,% UIFO %.-,%

    View full-size slide

  34. w 5BSHFUOPEF 3FT/FU

    w 7BOJMMBNPEFM
    #FTULOPXMFEHFUSBOTGFSHSBQI OPEFT

    த෦େֶϩΰ
    த෦େֶϩΰ

    ɹ%FFQ.VUVBM-FBSOJOHɹ
    "VYJMJBSZOPEF
    QSFUSBJOFE

    "VYJMJBSZOPEF
    5BSHFUOPEF
    5FBDIFSOFUXPSL
    'JSTU ,% UIFO %.-,%

    View full-size slide

  35. w %BUBTFUT$*'"3
    w /PPGOPEFT
    w 5BSHFUOPEF3FT/FU
    &YQFSJNFOUBMSFTVMUT
    த෦େֶϩΰ
    த෦େֶϩΰ

    .FUIPE 5PQDDDVSBDZ<> .PEFMPGBVYJMJBSZOPEF
    *OEFQFOEFOU r
    ,% 3FT/FU QSFUSBJOFE

    %.- 3FT/FU 3FT/FU
    0VST 3FT/FU QSFUSBJOFE
    3FT/FU

    View full-size slide

  36. 7JTVBMJ[BUJPOPGLOPXMFEHFUSBOTGFSHSBQI $*'"3

    த෦େֶϩΰ
    த෦େֶϩΰ

    OPEFT


    OPEFT


    OPEFT


    OPEFT


    OPEFT


    OPEFT


    *OEFQFOEFOU3FT/FU

    View full-size slide

  37. 7JTVBMJ[BUJPOPGLOPXMFEHFUSBOTGFSHSBQI $*'"3

    த෦େֶϩΰ
    த෦େֶϩΰ

    *OEFQFOEFOU3FT/FU

    OPEFT

    View full-size slide

  38. %FFQFOTFNCMFMFBSOJOHVTJOHLOPXMFEHFUSBOTGFSHSBQI<&$$7>
    த෦େֶϩΰ
    த෦େֶϩΰ

    View full-size slide

  39. w &YUFOEJOHLOPXMFEHFUSBOTGFSHSBQIGPSEFFQFOTFNCMFMFBSOJOH
    w .BYJNJ[JOHUIFBDDVSBDZPGUIFFOTFNCMFOPEFUPUSBJOCFUUFSFOTFNCMFOFUXPSLT
    "UUFOUJPOMPTTBUUFOUJPOMPTTJTBEEFEUPFBDIFEHF
    &OTFNCMFOPEFBNFDIBOJTNUPFOTFNCMFUIFPVUQVUPGFBDIOPEF
    %FFQFOTFNCMFMFBSOJOHVTJOH,5(<0LBNPUP &$$7>
    த෦େֶϩΰ
    த෦େֶϩΰ

    &OTFNCMFOPEF
    ^
    𝑦
    𝐿
    ^
    𝑦
    ,
    𝑒
    𝑒𝑛𝑠
    𝑚
    1
    𝑚
    2
    𝑚
    3
    ^
    𝑦
    ^
    𝑦
    ^
    𝑦
    𝐿
    ^
    𝑦
    ,3
    𝐿
    ^
    𝑦
    ,2
    𝐿
    ^
    𝑦
    ,1
    𝐿
    2,1
    𝐿
    1,2
    𝐿
    3,1
    𝐿
    1,3
    𝐿
    2,3
    𝐿
    3,2
    𝐿
    i,j
    =
    𝐾 𝐿
    (
    𝒑
    i
    ,
    𝒑
    j) ±
    𝐿
    𝐴
    𝑇
    (
    𝑸
    i
    ,
    𝑸
    j
    )
    "UUFOUJPOMPTT

    View full-size slide

  40. w "UUFOUJPOMPTTGSPNOPEFUPOPEF
    1FSGPSNTUPCSJOHPSUPTFQBSBUFUIFBUUFOUJPONBQTCFUXFFOUXPNPEFMT
    "UUFOUJPOMPTT
    த෦େֶϩΰ
    த෦େֶϩΰ

    4PVSDF %FTUJOBUJPO
    𝑚
    2
    𝑚
    1
    p2
    (c|x) p1
    (c|x)
    'PSXBSE
    𝑚
    1
    𝑚
    2
    𝑚
    3
    ^
    𝑦
    ^
    𝑦
    ^
    𝑦
    𝐿
    ^
    𝑦
    ,3
    𝐿
    ^
    𝑦
    ,2
    𝐿
    ^
    𝑦
    ,1
    𝐿
    2,1
    𝐿
    1,2
    𝐿
    3,1
    𝐿
    1,3
    𝐿
    2,3
    𝐿
    3,2
    ,-EJW
    x x
    "UUFOUJPO "UUFOUJPO
    𝑸
    =
    𝐶

    𝑖
    =1
    𝑨
    𝑖
    𝑝
    ɿGFBUVSFNBQT
    ɿDIBOOFMT
    ɿOPSN
    𝑨
    𝑖
    𝐶
    𝑝
    ,--PTT
    𝐿
    𝐴𝑇
    𝐿
    𝐴𝑇
    (
    𝑸
    2
    ,
    𝑸
    1
    ) =
    1
    𝐽
    𝐽

    𝑗
    𝑸
    𝑗
    2
    𝑸
    𝑗
    2
    2

    𝑸
    𝑗
    1
    𝑸
    𝑗
    1
    2
    2
    "UUFOUJPOMPTT
    (BUF
    𝐿
    2,1
    = G(
    𝐾𝐿
    (
    𝒑
    2
    ,
    𝒑
    1) ±
    𝐿
    𝐴 𝑇
    (
    𝑸
    2
    ,
    𝑸
    1
    ))
    CSJOHJOH
    TFQBSBUJOH
    +

    View full-size slide

  41. w 6QEBUFOPEF CZCBDLQSPQBHBUJPOVTJOH,-EJWFSHFODFBOEBUUFOUJPOMPTT
    𝑚
    1
    )PXUPUSBOTGFSLOPXMFEHFGSPNOPEFUPOPEF
    த෦େֶϩΰ
    த෦େֶϩΰ

    𝑚
    2
    𝑚
    1
    𝑚
    1
    𝑚
    2
    𝑚
    3
    ^
    𝑦
    ^
    𝑦
    ^
    𝑦
    𝐿
    ^
    𝑦
    ,3
    𝐿
    ^
    𝑦
    ,2
    𝐿
    ^
    𝑦
    ,1
    𝐿
    1,2
    𝐿
    3,1
    𝐿
    1,3
    𝐿
    2,3
    𝐿
    3,2
    x x
    4PVSDF %FTUJOBUJPO
    𝐿
    2,1
    #BDLXBSE
    𝐿
    2,1
    = G(
    𝐾𝐿
    (
    𝒑
    2
    ,
    𝒑
    1) ±
    𝐿
    𝐴 𝑇
    (
    𝑸
    2
    ,
    𝑸
    1
    ))
    ,-EJW
    𝐿
    𝐴𝑇
    (BUF
    "UUFOUJPO "UUFOUJPO
    Detach
    Back-prop
    CSJOHJOH
    TFQBSBUJOH
    +

    View full-size slide

  42. w .BYJNJ[JOHUIFBDDVSBDZPGUIFFOTFNCMFOPEFBTBUBSHFUOPEF
    &OTFNCMFNFDIBOJTNDBMDVMBUFTUIFBWFSBHFPVUQVUPGBMMOPEFT
    &OTFNCMFOPEF
    த෦େֶϩΰ
    த෦େֶϩΰ

    ^
    𝑦
    𝐿
    ^
    𝑦
    ,
    𝑒
    𝑒𝑛𝑠
    𝑚
    1
    𝑚
    2
    𝑚
    3
    ^
    𝑦
    ^
    𝑦
    ^
    𝑦
    𝐿
    ^
    𝑦
    ,3
    𝐿
    ^
    𝑦
    ,2
    𝐿
    ^
    𝑦
    ,1
    𝐿
    2,1
    𝐿
    1,2
    𝐿
    3,1
    𝐿
    1,3
    𝐿
    2,3
    𝐿
    3,2
    𝑚
    1
    𝑚
    2
    𝑒𝑛𝑠
    &OTFNCMFOPEF
    𝑚
    3
    &OTFNCMFNFDIBOJTN
    𝑝
    (
    𝑐 𝑥
    ) =
    𝑝
    1
    (
    𝑐 𝑥
    ) +
    𝑝
    2
    (
    𝑐 𝑥
    ) +
    𝑝
    3
    (
    𝑐
    |
    𝑥
    )
    𝑝
    1
    (
    𝑐 𝑥
    )
    𝑝
    2
    (
    𝑐 𝑥
    )
    𝑝
    3
    (
    𝑐 𝑥
    )
    𝑝
    (
    𝑐 𝑥
    )

    View full-size slide

  43. &
    ff
    FDUJWFOFTTPGFOTFNCMFDPMMBCPSBUJOHMFBSOJOH
    த෦େֶϩΰ
    த෦େֶϩΰ

    ˠ&OTFNCMFBDDVSBDZJNQSPWFEJOLOPXMFEHFUSBOTGFSHSBQI
    5PUBMOVNCFSPGQBSBNFUFST
    5PQBDDVSBDZ<>
    4UBOGPSE%PHTEBUBTFU

    View full-size slide

  44. #FTUHSBQIGPSFOTFNCMF OPEFT

    த෦େֶϩΰ
    த෦େֶϩΰ

    ˠ"DRVJSFEJ
    ff
    FSFOUBUUFOUJPONBQT EJWFSTJUZTVJUBCMFGPSFOTFNCMFT

    JOQVU

    View full-size slide

  45. #FTUHSBQIGPSFOTFNCMF OPEFT

    த෦େֶϩΰ
    த෦େֶϩΰ

    ˠ"DRVJSFEJ
    ff
    FSFOUBUUFOUJPONBQT EJWFSTJUZTVJUBCMFGPSFOTFNCMFT

    JOQVU
    #SJOHDMPTFSUP
    #SJOHDMPTFSFBDIPUIFS
    #SJOHDMPTFSUP
    #SJOHDMPTFSUP
    #SJOHDMPTFSUP
    4FQBSBUFGSPN
    4FQBSBUFGSPN

    View full-size slide

  46. ,OPXMFEHFEJTUJMMBUJPOGSPNFOTFNCMFNPEFMT
    த෦େֶϩΰ
    த෦େֶϩΰ

    w ,%GSPNFOTFNCMFNPEFMBTUFBDIFS
    5FBDIFSOFUXPSLFOTFNCMFNPEFMPGPQUJNJ[FEHSBQI 3FO/FUº

    4UVEFOUOFUXPSL3FT/FU
    *OQVU
    𝒙
    4UVEFOU/FUXPSL
    /FUXPSL
    𝑚
    1
    𝒍
    1
    (
    𝒙
    )
    𝒍
    3
    (
    𝒙
    )
    /FUXPSL
    𝑚
    3
    𝒍
    𝑒
    𝑛
    𝑠
    (
    𝒙
    )
    𝒑
    𝑠
    (
    𝒙
    ) $PSSFDUMBCFM ^
    𝑦
    ,OPXMFEHFUSBOTGFS
    𝒑
    𝑒
    𝑛
    𝑠
    (
    𝒙
    )
    4PGUNBY
    5FBDIFS/FUXPSL
    𝑒 𝑛 𝑠
    𝑚
    1
    𝑚
    2
    𝑚
    3
    ,OPXMFEHF5SBOTGFS
    (SBQI

    View full-size slide

  47. ,OPXMFEHFEJTUJMMBUJPOGSPNFOTFNCMFNPEFMT
    த෦େֶϩΰ
    த෦େֶϩΰ

    )JHISFDPHOJUJPOQFSGPSNBODFXJUIUIFTBNFOVNCFSPGQBSBNFUFSTCZ
    VTJOHFOTFNCMFNPEFMCBTFEPOUIFLOPXMFEHFUSBOTGFSHSBQI
    5PUBMOVNCFSPGQBSBNFUFST
    5PQBDDVSBDZ<>
    ,OPXMFEHFEJTUJMMBUJPO
    ,OPXMFEHFEJTUJMMBUJPO
    4UBOGPSE%PHTEBUBTFU
    4UVEFOUOFUXPSL
    3FT/FU

    4UVEFOUOFUXPSL
    3FT/FU

    View full-size slide

  48. $PODMVTJPO
    த෦େֶϩΰ
    த෦େֶϩΰ

    w ,OPXMFEHF5SBOTGFS(SBQI
    "OPWFMHSBQISFQSFTFOUBUJPOPGLOPXMFEHFUSBOTGFSGPSEFFQDPMMBCPSBUJWFMFBSOJOH
    'PVSHBUFGVODUJPOTDPOUSPMIPXUIFLOPXMFEHFJTUSBOTGFSSFE
    4FBSDIJOHGPSCFTULOPXMFEHFUSBOTGFSHSBQICZVTJOHIZQFSQBSBNFUFSTFBSDI
    w 8IBUXFGPVOECZ,OPXMFEHF5SBOTGFS(SBQIGPS%$-
    'VTJPOPG,%BOE%.-JNQSPWFTQFSGPSNBODF
    %FFQDPMMBCPSBUJOHMFBSOJOHCBTFEPO,%BOE%.-

    View full-size slide

  49. (JU)VC
    த෦େֶϩΰ
    த෦େֶϩΰ

    ,OPXMFEHF5SBOTGFS(SBQI
    IUUQTHJUIVCDPNTPNBNJOBNJ%$-
    5BLBZPTIJ:BNBTIJUB
    4PNB.JOBNJ 5TVCBTB)JSBLBXB
    $PMMBCPSBUPST
    /BPLJ0LBNPUP

    View full-size slide