Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What Can Neural Networks Reason About?

What Can Neural Networks Reason About?

Scatter Lab Inc.

May 29, 2020
Tweet

More Decks by Scatter Lab Inc.

Other Decks in Research

Transcript

  1. 8IBUJT3FBTPOJOH *OUSPEVDUJPO3FBTPOJOH • ୶ܻ: ঌҊ ੓ח Ѫਵ۽ࠗఠ ঌ૑ ޅೞח Ѫਸ

    ࢎҊೣ • ীಘ఑਷ ౵ܻী ੓যਃ -> ౵ܻח ೐یझ੄ ࣻبীਃ -> ীಘ఑਷ যו աۄী ੓ਸө? • ੿ࠁܳ ઱঻ਸ ٸ, Ӓ ੿ࠁ۽ࠗఠ ҙ଴ೞ૑ ޅೠ Ѫী ؀ೠ ࢜۽਍ ੿ࠁܳ ب୹೧ղח ੘স
  2. 8IBUJT3FBTPOJOH *OUSPEVDUJPO3FBTPOJOH • ੿ࠁܳ ઱঻ਸ ٸ, Ӓ ੿ࠁ۽ࠗఠ ҙ଴ೞ૑ ޅೠ

    Ѫী ؀ೠ ࢜۽਍ ੿ࠁܳ ب୹೧ղח ੘স • न҃ݎ੄ ୶ܻ ޙઁ: ੿ࠁ/ࣁ࢚ਸ ҳઑചೞҊ Ӓ ҳઑ۽ࠗఠ Ѿҗܳ ৘ஏೞب۾ ೟णदఇ • GNN, Neural Symbolic Programs (Semantic Parsing), Deep Sets
  3. %FGJOJUJPOPG3FBTPOJOH *OUSPEVDUJPO3FBTPOJOH • ޛ୓ ૘೤ s ∈ Sо ੓যࢲ п

    sܳ Xۄח ߭ఠ۽ ಴അೡ ࣻ ੓׮Ҋ о੿ • ࢚ട {S1, S2, S3, …} ী ؀೧ࢲ ੿׹ ۄ߰ {y1, y2, y3, …} о ੓ਸ ٸ • ੐੄੄ ࠁ૑ ޅೠ ࢚ട Sী ؀ೠ ੿׹ ۄ߰ yܳ ب୹ೞח ೣࣻ y=g(S)ܳ ঳ח Ѫ੉ ݾ಴੐
  4. 4USVDUVSFTGPS3FBTPOJOH *OUSPEVDUJPO3FBTPOJOH • Deep Sets • S = {ࡈр ҕ,

    ౵ۆ ҕ, ֢ۆ ҕ} ੉ۄҊ о੿೧ࠁݶ, • ࢎۈ ੑ੢ীࢲח ࣽࢲо ࢚ҙ হ਺ {ࡈр ҕ, ౵ۆ ҕ, ֢ۆ ҕ} = {౵ۆ ҕ, ֢ۆ ҕ, ࡈр ҕ} • Permutation Invariant ೞѱ ೣࣻ gܳ ࢸ҅ೠ Ѫ੉ Deep set੐ • য۰ਕ ࠁ੉૑݅ Ӓր MLP ೤੄ MLPۄҊ ࢤп • যڃ ࣽࢲ۽ ৬ب ࢚ҙ হ਺
  5. ,JOEPG3FBTPOJOH *OUSPEVDUJPO3FBTPOJOH • ੷੗ٜ਷ ୶ܻ ޙઁܳ դ੉ب৬ ౠ૚ী ٮۄࢲ ֎

    о૑۽ ࠙ܨೣ • ా҅ ਃড (Summary Statistics): ઱য૓ ੑ۱ਸ ‘ࣁח’ ޙઁ • ৘द) ز੹੉ ࢎ૓ী ݻ ѐ ੓ա? Ҋন੉ی ѐ઺ী যڃ Ѫ੉ ݆ա? • ҙ҅ ࢲৌ (Relational Argmax): ઱য૓ ੑ۱ীࢲ ࢚؀੸ ҙ҅ܳ ٮ૑ח ޙઁ • ৘द) о੢ ݣܻ ڄযઉ ੓ח ޛ୓ ह਷ যڃ ࢝ӭੋо? • ز੸ ೐۽Ӓې߁ (Dynamic Programming): Ѿҗܳ ੷੢ೞݴ ಹח ޙઁ • ৘द) ೖࠁա஖, ઁੌ ૣ਷ ӡ ଺ӝ • NP-Hard Problem: ??? • ৘द) ઱য૓ ૘೤ী ؀೧ࢲ Ӓ ೤੉ 0੉غח ࠗ࠙ ૘೤ਸ ଺ਸ ࣻ ੓ਸө?
  6. ,JOEPG3FBTPOJOH *OUSPEVDUJPO3FBTPOJOH • ੷੗ٜ਷ ୶ܻ ޙઁܳ դ੉ب৬ ౠ૚ী ٮۄࢲ ֎

    о૑۽ ࠙ܨೣ • ా҅ ਃড (Summary Statistics): ઱য૓ ੑ۱ਸ ‘ࣁח’ ޙઁ • ৘द) ز੹੉ ࢎ૓ী ݻ ѐ ੓ա? Ҋন੉ی ѐ઺ী যڃ Ѫ੉ ݆ա? • ҙ҅ ࢲৌ (Relational Argmax): ઱য૓ ੑ۱ীࢲ ࢚؀੸ ҙ҅ܳ ٮ૑ח ޙઁ • ৘द) о੢ ݣܻ ڄযઉ ੓ח ޛ୓ ह਷ যڃ ࢝ӭੋо? • ز੸ ೐۽Ӓې߁ (Dynamic Programming): Ѿҗܳ ੷੢ೞݴ ಹח ޙઁ • ৘द) ೖࠁա஖, ઁੌ ૣ਷ ӡ ଺ӝ • NP-Hard Problem: ??? • ৘द) ઱য૓ ૘೤ী ؀೧ࢲ Ӓ ೤੉ 0੉غח ࠗ࠙ ૘೤ਸ ଺ਸ ࣻ ੓ਸө?
  7. ,JOEPG3FBTPOJOH *OUSPEVDUJPO3FBTPOJOH • ੷੗ٜ਷ ୶ܻ ޙઁܳ դ੉ب৬ ౠ૚ী ٮۄࢲ ֎

    о૑۽ ࠙ܨೣ • ా҅ ਃড (Summary Statistics): ઱য૓ ੑ۱ਸ ‘ࣁח’ ޙઁ • ৘द) ز੹੉ ࢎ૓ী ݻ ѐ ੓ա? Ҋন੉ی ѐ઺ী যڃ Ѫ੉ ݆ա? • ҙ҅ ࢲৌ (Relational Argmax): ઱য૓ ੑ۱ীࢲ ࢚؀੸ ҙ҅ܳ ٮ૑ח ޙઁ • ৘द) о੢ ݣܻ ڄযઉ ੓ח ޛ୓ ह਷ যڃ ࢝ӭੋо? • ز੸ ೐۽Ӓې߁ (Dynamic Programming): Ѿҗܳ ੷੢ೞݴ ಹח ޙઁ • ৘द) ೖࠁա஖, ઁੌ ૣ਷ ӡ ଺ӝ • NP-Hard Problem: ??? • ৘द) ઱য૓ ૘೤ী ؀೧ࢲ Ӓ ೤੉ 0੉غח ࠗ࠙ ૘೤ਸ ଺ਸ ࣻ ੓ਸө?
  8. ,JOEPG3FBTPOJOH *OUSPEVDUJPO3FBTPOJOH • ੷੗ٜ਷ ୶ܻ ޙઁܳ դ੉ب৬ ౠ૚ী ٮۄࢲ ֎

    о૑۽ ࠙ܨೣ • ా҅ ਃড (Summary Statistics): ઱য૓ ੑ۱ਸ ‘ࣁח’ ޙઁ • ৘द) ز੹੉ ࢎ૓ী ݻ ѐ ੓ա? Ҋন੉ی ѐ઺ী যڃ Ѫ੉ ݆ա? • ҙ҅ ࢲৌ (Relational Argmax): ઱য૓ ੑ۱ীࢲ ࢚؀੸ ҙ҅ܳ ٮ૑ח ޙઁ • ৘द) о੢ ݣܻ ڄযઉ ੓ח ޛ୓ ह਷ যڃ ࢝ӭੋо? • ز੸ ೐۽Ӓې߁ (Dynamic Programming): Ѿҗܳ ੷੢ೞݴ ಹח ޙઁ • ৘द) ೖࠁա஖, ઁੌ ૣ਷ ӡ ଺ӝ • NP-Hard Problem: ࠺Ѿ੿੸ ౚ݂ӝ҅о ׮೦दрী ಽ ࣻ ੓ח ޙઁܳ ೧׼ ޙઁٜ۽ ׮೦ दрী ജਗೡ ࣻ ੓ח ૘೤ • ৘द) ઱য૓ ૘೤ী ؀೧ࢲ Ӓ ೤੉ 0੉غח ࠗ࠙ ૘೤ਸ ଺ਸ ࣻ ੓ਸө?
  9. ,JOEPG3FBTPOJOH *OUSPEVDUJPO3FBTPOJOH • ੷੗ٜ਷ п ޙઁٜী ؀೧ࢲ যڃ न҃ݎ ҳઑ۽

    ಽ ࣻ ੓ח૑ ׹ೞҊ੗ ೞ৓਺ • ా҅ ਃড (Summary Statistics): ઱য૓ ੑ۱ਸ ‘ࣁח’ ޙઁ • ৘द) GNNҗ Deep Setਵ۽ח ੜ ಽ ࣻ ੓૑݅, MLP۽ח ಽӝ য۰਑ • ҙ҅ ࢲৌ (Relational Argmax): ઱য૓ ੑ۱ীࢲ ࢚؀੸ ҙ҅ܳ ٮ૑ח ޙઁ • ৘द) GNNਵ۽ח ಽ ࣻ ੓૑݅, Deep Setਵ۽ח ಽӝ য۰਑ • ز੸ ೐۽Ӓې߁ (Dynamic Programming): Ѿҗܳ ੷੢ೞݴ ಹח ޙઁ • ৘द) GNNਵ۽ ಽ ࣻ ੓਺ • NP-Hard Problem: ࠺Ѿ੿੸ ౚ݂ӝ҅о ׮೦दрী ಽ ࣻ ੓ח ޙઁܳ ೧׼ ޙઁٜ۽ ׮೦ दрী ജਗೡ ࣻ ੓ח ૘೤ • ৘द) Exhaustive Searchо ೙ਃೣ
  10. 4USVDUVSF$PNQMFYJUZPG1SPCMFN "MHPSJUINJD"MJHONFOU • ޙઁ৬ Ӓ ޙઁܳ ಹח न҃ݎ ҳઑܳ о੿೧ࠁӝ

    • ݅ডী ޙઁ੄ ҳઑܳ न҃ݎ੄ ҳઑী ؀਽दఆ ࣻ ੓׮ݶ?
  11. 4USVDUVSF$PNQMFYJUZPG1SPCMFN "MHPSJUINJD"MJHONFOU • ޙઁ ಽ੉੄ ҙ੼ীࢲ ֤ޙਸ ׮਺ਸ ੹ઁೣ •

    ೧ࢳ: • ׮਺ ઑѤਸ ݅઒ೞח ޙઁח GNNਵ۽ ؀਽दఆ ࣻ ੓׮ • Ӓ ޙઁী աఋաח ޛ୓ח ୭؀ Nѐө૑݅ оמೞҊ, • Ӓ ޙઁ੄ ޛ୓੄ ࣽࢲח ޙઁܳ ಹחؘ ޖҙೞ׮
  12. 4USVDUVSF$PNQMFYJUZPG1SPCMFN "MHPSJUINJD"MJHONFOU • ޙઁ ಽ੉੄ ҙ੼ীࢲ ֤ޙਸ ׮਺ਸ ੹ઁೣ •

    ೧ࢳ: • ׮਺ ઑѤਸ ݅઒ೞח ޙઁח GNNਵ۽ ؀਽दఆ ࣻ ੓׮ • ӒܻҊ ੐੄੄ GNN਷ MLP۽ ؀਽दఆ ࣻ ੓׮
  13. 4USVDUVSF$PNQMFYJUZPG1SPCMFN "MHPSJUINJD"MJHONFOU • ޙઁ ಽ੉੄ ҙ੼ীࢲ ֤ޙਸ ׮਺ਸ ੹ઁೣ •

    ೧ࢳ: • ׮਺ ઑѤਸ ݅઒ೞח ޙઁח GNNਵ۽ ؀਽दఆ ࣻ ੓׮ • ӒܻҊ ੐੄੄ GNN਷ MLP۽ ؀਽दఆ ࣻ ੓׮ • ޙઁ: “ޙઁ ೟ण”੄ ҃਋ب Ӓۡө???
  14. 4USVDUVSF$PNQMFYJUZPG1SPCMFN "MHPSJUINJD"MJHONFOU • ޙઁ : ޙઁ ೟णী ؀೧ࢲب Ӓۡө? •

    ೧ࠁݶ MLPח ੜ ޅ ߓ਋ח Ѫ э਺: ҳઑ੸ਵ۽ ޙઁܳ ୶࢚ചೞח מ۱੉ ࠗ઒೧ࢲ • GNN਷ п ױ҅ܳ ߓ਎ ࣻ ੓૑݅, MLPۄݶ ܖ೐ ੹୓੄ Ѿҗޛਸ ೞա۽ ߓਕঠ ೣ
  15. 1"$-FBSOBCJMJUZBOE"MJHONFOU "MHPSJUINJD"MJHONFOU • Probably approximately correct learning (Ѣ੄ ੿ഛೠ ೟ण)

    • ੑ۱җ ୹۱हਸ ࢠ೒݂೧ࢲ ೟णೞח ঌҊ્ܻਸ ࢤп೧ࠁݶ • Probably: ֫਷ ഛܫ۽ • Approximately Correct: ݽ؛੄ ী۞о Threshold ޷݅ੌѢ׮
  16. 1"$-FBSOBCJMJUZ "MHPSJUINJD"MJHONFOU • Probably approximately correct learning (Ѣ੄ ੿ഛೠ ೟ण)

    • ೧ࢳ: • ਋ܻ ݽ؛੉ ࢠ೒݂ਸ ా೧ࢲ ೟णदఆ ࣻ ੓׮Ҋ о੿೧ࠁ੗ (PAC ઑѤ) • Ӓ۞ݶ ী۞بܳ ઁೠೞӝ ਤ೧ࢲח ݆੉ ࡳইঠ ೠ׮ (୭ࣗ M ݅ఀ੄ ࠂ੟ب۽ ࡳইঠೣ)
  17. 1"$-FBSOBCJMJUZ "MHPSJUINJD"MJHONFOU • Probably approximately correct learning (Ѣ੄ ੿ഛೠ ೟ण)

    • ؊ ए਍ ೧ࢳ: • ࢠ೒݂ਵ۽ ೟णदఆ ٸ, ࢿמ ֫੉۰ݶ ݆੉ ࡳইঠೠ׮!
  18. 1"$-FBSOBCJMJUZ&YBNQMF "MHPSJUINJD"MJHONFOU • MLPח ݽٚ ੄ࢎѾ੿ җ੿ਸ ೞա੄ ܖ೐۽ р઱ೡ

    ࣻ ߆ী হਵ޲۽ • ೧ࢳ: • MLP۽ ޙઁܳ ಽѱ दః۰ݶ • ୭ࣗೠ ୐ Layer੄ ௼ӝ ী ؀೧ ੑ۱ ରਗ੄ ௼ӝ ݅ఀ Ѣٟઁғೠ Ѫ • ӒѪਸ فߣ૩ Layer੄ ௼ӝী ؀೧ ғೠ Ѫ • ݅ఀ਷ ࢠ೒݂೧ঠ ೟णदఆ ࣻ ੓ਸ Ѫ
  19. 1"$-FBSOBCJMJUZBOE"MJHONFOU "MHPSJUINJD"MJHONFOU • ݽ؛੉ PAC ള۲ оמ ઑѤਸ ݅઒ೠ׮ݶ ׮਺੄

    Alignment ܳ ࣻधਵ۽ ॶ ࣻ ੓਺ • ೧ࢳ: • ੐੄੄ ޙઁী ؀೧ࢲ Ӓ ޙઁܳ f1, f2, f3,..ਵ۽ ଂѓ ࣻ ੓׮Ҋ о੿ೞҊ • Ӓ ଂѐ૓ ٜࠗ࠙ਸ пп੄ न҃ݎਵ۽ ؀਽दఆ ࣻ ੓׮Ҋ о੿ೞݶ • Alignment ػ׮ח Ѫ਷ • пп ݽ؛ਸ ള۲दఆ ٸ, ӝઓী Mѐ ࡳ਷ Ѫࠁ׮ ੸ѱ ࡳইب غח ҃਋
  20. 1"$-FBSOBCJMJUZBOE"MJHONFOU "MHPSJUINJD"MJHONFOU • ݽ؛੉ PAC ള۲ оמ ઑѤਸ ݅઒ೠ׮ݶ ׮਺੄

    Alignment ܳ ࣻधਵ۽ ॶ ࣻ ੓਺ • ؊ ए਍ ೧ࢳ: • য۰਍ ޙઁܳ ए਍ ࣁࠗ ޙઁ۽ ଂѐࢲ ೟णदௌਸ ٸ, • Ӓ ए਍ ࣁࠗ ޙઁ пп੉ ੸਷ ؘ੉ఠ۽ب ੜ ߓ਎ ࣻ ੓঻ਵݶ જѷ׮!
  21. #FUUFS"MJHONFOU#FUUFS(FOFSBMJ[BUJPO "MHPSJUINJD"MJHONFOU • Alignmentܳ ੜ ೡ ࣻ ੓ח ݽ؛਷ ੜ

    ߓ਎ ࣻ ੓਺ • ппਸ ੜ ଂѐࢲ ߓ਋ח ੘স੉ ੜ ഛ݀ػ׮ݶ Ӓۧѱ ػ׮ח ܻࣗ • ցޖ ׼োೠ ݈ੋ٠ • ਋ܻ੄ ҙ੼ীࢲ BERTо MLP ա LSTM ࠁ׮ ੜೞח ੉ਬח? • BERT ҳઑо ޙ੢੄ ਫ਼੤੸ੋ ҳઑܳ ؊ Generalization ਸ ੜ ೮ӝ ٸޙ
  22. #FUUFS"MJHONFOU#FUUFS(FOFSBMJ[BUJPO "MHPSJUINJD"MJHONFOU • Generalization ਸ ੜ ޅೞח ݽ؛਷ য۰਍ ޙઁܳ

    ೟णೞӝ য۰਑ • ࣘغѱ ݈ೞ੗ݶ ݽ؛੄ ࠂ੟بী ٮۄ Ӓ ޙઁܳ ಽ “ल”੉ ࠁੋ׮ח Ѫ • п ޙઁী ؀ೠ ࢿמ
  23. #FUUFS"MJHONFOU#FUUFS(FOFSBMJ[BUJPO "MHPSJUINJD"MJHONFOU • Generalization ਸ ੜ ޅೞח ݽ؛਷ য۰਍ ޙઁܳ

    ೟णೞӝ য۰਑ • ࣘغѱ ݈ೞ੗ݶ ݽ؛੄ ࠂ੟بী ٮۄ Ӓ ޙઁܳ ಽ “ल”੉ ࠁੋ׮ח Ѫ • Monster Trainer (Path Searching) ޙઁী ؀ೠ п ݽ؛੄ ࢿמ
  24. $PODMVTJPO $PODMVTJPO • যڃ न҃ݎ ݽ؛ਸ ഝਊ೧ࢲ যڃ ޙઁী ؀೧ࢲ

    ಽ ٸ • Ӓ ݽ؛੄ ҳઑо ޙઁ੄ ࢲࢎ ҳઑܳ ನҚೡ ࣻ ੓যঠ ೣ • ݅ড Ӓۧ૑ ঋ׮ݶ ই઱ ݆਷ ࢠ೒੉ ೙ਃೣ = ݽ؛੉ ޙઁܳ ੌ߈ചೡ ࣻ হ਺ • ࠂ੟ೠ ݽ؛੉ ԙ ੜ ಽ૑ח ঋ૑݅, рױೠ ݽ؛਷ ಽӝ য۰਑ • ਋ܻীѱ ઱ח दࢎ • औѱ ߓ਍ ݽ؛, рױೠ ഋక۽ ҳࢿػ ݽ؛੉ ঱য੄ ޷ޑೣਸ ੜ աఋյ ࣻ ੓ਸө? • അ੤੄ BERT ҳઑח ցޖ ࠂ੟ೠ Ѫ੉ ইקө?