Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Paper Reading: Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics

Paper Reading: Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics

Kento Nozawa

June 12, 2018
Tweet

More Decks by Kento Nozawa

Other Decks in Research

Transcript

  1. Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural

    Image Statistics Student seminar @ Sugiyama-Sato-Honda Lab D1, DC1 @ The University of Tokyo Kento NOZAWA (@nzw0301) Michael U. Gutmann, Aapo Hyvärinen, JMLR, 2012.
  2. Why I choose this paper? • Useful approximation to calculate

    an intractable partition function in probabilistic models • Fast and simple to learn probabilistic models • Sometimes used in recent DNN/ ML papers 2
  3. or . Z(✓) is A sum/integral is intractable over unnormalized

    
 probability distribution for many models. Goal: Estimation parameters in a probabilistic model: Background: Probabilistic Models 3 ✓ <latexit sha1_base64="317VWb7h3beDW2lJpPQYuAuNpOg=">AAACaXichVFNLwNBGH66vqo+2nIRLqIhTs27LkQiGi6O2mpJENldg9Xt7mZ32qQaibOTm+BEIiJ+hgN/wKE/QXokcXHwdttEELyTmXnmmfd555kZ3bVMXxLVQkpbe0dnV7g70tPb1x+NxQfyvlPyDJEzHMvx1nTNF5Zpi5w0pSXWXE9oRd0Sq3phsbG/Whaebzr2iqy4YrOo7drmjmlokqn8htwTUtuKJShJQYz+BGoLJOYfInNHAJad2A02sA0HBkooQsCGZGxBg89tHSoILnObqDLnMTKDfYFDRFhb4izBGRqzBR53ebXeYm1eN2r6gdrgUyzuHitHMU5PdEsv9Eh39Ezvv9aqBjUaXio8602tcLeix0PZt39VRZ4l9j5Vf3qW2MFM4NVk727ANG5hNPXlg9OX7GxmvDpBV1Rn/5dUo3u+gV1+Na7TInOBCH+A+v25f4L8VFKlpJqmRGoBzQhjBGOY5PeeRgpLWEaOz93HCc5wHqorcWVIGW6mKqGWZhBfQkl8APjwje0=</latexit> <latexit sha1_base64="G5x5alePLF85znahbXBopKC7Dq4=">AAACaXichVFNLwNBGH66vtdXy0W4NBri1LzrQiRCuDi2aDUpkd01WLa7m91pk2r8ASc3wQkRET/DgT8g0p8gjiQuDt5umwiCdzIzzzzzPu88M2N4thVIompEaWpuaW1r71A7u7p7eqOxvmzgFn1TZEzXdv2coQfCthyRkZa0Rc7zhV4wbLFi7M7X9ldKwg8s11mWZU+sFfQtx9q0TF0ylV2V20Lq69EEJSmM+E+gNUBi5k6d9i4e1ZQbvcIqNuDCRBEFCDiQjG3oCLjloYHgMbeGCnM+IyvcF9iHytoiZwnO0Jnd5XGLV/kG6/C6VjMI1SafYnP3WRnHCD3QNb3QPd3QE73/WqsS1qh5KfNs1LXCW+89GFh6+1dV4Fli+1P1p2eJTUyGXi327oVM7RZmXV/aO3pZmlocqYzSOT2z/zOq0i3fwCm9mpdpsXgKlT9A+/7cP0F2PKlRUktTYnYO9WjHEIYxxu89gVksIIUMn7uDQxzjJPKsxJQBZbCeqkQamn58CSXxAZ6+j2E=</latexit> <latexit sha1_base64="G5x5alePLF85znahbXBopKC7Dq4=">AAACaXichVFNLwNBGH66vtdXy0W4NBri1LzrQiRCuDi2aDUpkd01WLa7m91pk2r8ASc3wQkRET/DgT8g0p8gjiQuDt5umwiCdzIzzzzzPu88M2N4thVIompEaWpuaW1r71A7u7p7eqOxvmzgFn1TZEzXdv2coQfCthyRkZa0Rc7zhV4wbLFi7M7X9ldKwg8s11mWZU+sFfQtx9q0TF0ylV2V20Lq69EEJSmM+E+gNUBi5k6d9i4e1ZQbvcIqNuDCRBEFCDiQjG3oCLjloYHgMbeGCnM+IyvcF9iHytoiZwnO0Jnd5XGLV/kG6/C6VjMI1SafYnP3WRnHCD3QNb3QPd3QE73/WqsS1qh5KfNs1LXCW+89GFh6+1dV4Fli+1P1p2eJTUyGXi327oVM7RZmXV/aO3pZmlocqYzSOT2z/zOq0i3fwCm9mpdpsXgKlT9A+/7cP0F2PKlRUktTYnYO9WjHEIYxxu89gVksIIUMn7uDQxzjJPKsxJQBZbCeqkQamn58CSXxAZ6+j2E=</latexit> <latexit sha1_base64="B4/KVo8DHTFQ13Zj30fSx7QNPAM=">AAACaXichVG7SgNBFD1Z3/GRRBvRJhgUq3DXRrESbSyNmgckIrvrmKzZ7C67k4AGf8DKTtRKQUT8DBt/wCKfICkVbCy82SyIinqHmTlz5p47Z2Z01zJ9SdSKKD29ff0Dg0PR4ZHRsVg8MZ7znbpniKzhWI5X0DVfWKYtstKUlii4ntBquiXyenWts59vCM83HXtbHrpip6aVbXPfNDTJVK4kK0Jqu/EUpSmI5E+ghiCFMDac+C1K2IMDA3XUIGBDMragwedWhAqCy9wOmsx5jMxgX+AYUdbWOUtwhsZslccyr4oha/O6U9MP1AafYnH3WJnELD3RHb3QI93TM73/WqsZ1Oh4OeRZ72qFuxs7mdx6+1dV41mi8qn607PEPpYCryZ7dwOmcwujq28cnb1sLW/ONufomtrs/4pa9MA3sBuvxk1GbF4iyh+gfn/unyC3kFYprWYotbIafsUgpjGDeX7vRaxgHRvI8rkHOMU5LiJtJaFMKlPdVCUSaibwJZTUB6pdjCA=</latexit> Partition function p(x; ✓) = 1 Z(✓) p0(x; ✓) <latexit sha1_base64="7gY4fYlpOkCxnEDZkQNamIz0Lr0=">AAACmXichVFNSxtBGH5cWz9iq6leBC9LQ0p6Ce+KoFSEqBfxpKbR0CSG3XWii5vdZXcS1CV/wD8g6KkFKaV/Quil7b2H/IOKBw8p9OLBN5tAq0F9h5l55pn3eeeZGcOzrUASNfuU/mfPBwaHhmMjL16OjsVfjW8Gbs03Rc50bdfPG3ogbMsROWlJW+Q9X+hVwxZbxv5ye3+rLvzAcp338tATpaq+61gVy9QlU+X4jJc6mFeLck9I/a26oBYrvm6GWiP8kOqSDTX0Gtuk/pdXjicoTVGovUDrgkQmc/37AsCaG/+MInbgwkQNVQg4kIxt6Ai4FaCB4DFXQsicz8iK9gUaiLG2xlmCM3Rm93nc5VWhyzq8btcMIrXJp9jcfVaqSNIv+kIt+k5f6ZJuHqwVRjXaXg55Njpa4ZXHjiezf59UVXmW2PunetSzRAVzkVeLvXsR076F2dHXj05a2XcbyfANfaIr9v+RmvSNb+DU/5jn62LjDDH+AO3+c/eCzem0RmltnX9iCZ0YwhReI8XvPYsMVrCGHJ97igv8wE9lSllUVpTVTqrS19VM4E4o2VtadJ9I</latexit> <latexit sha1_base64="Oa0xPtXVUZqDwpGLQAeyWNMf0Jk=">AAACmXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuYFGhUWCvElGSkliRqKtgqxKQVJSZXG9ZWR2lABWsVqgtq4wwUkNTFCygb6BmAgQImwxDKUHZweP9ij9gclYB8geUMMQwpDPkMyQylDLkMqQx5DCVAdg5DIkMxEEYzGDIYMBQAxWIZqoFiRUBWJlg+laGWgQuotxSoKhWoIhEomg0k04G8aKhoHpAPMrMYrDsZaEsOEBcBdSowqBpcNVhp8NnghMFqg5cGf3CaVQ02A+SWSiCdBNGbWhDP3yUR/J2grlwgXcKQgdCF180lDGkMFmC3ZgLdXgAWAfkiGaK/rGr652CrINVqNYNFBq+B7l9ocNPgMNAHeWVfkpcGpgbNZuACRoAhenBjMsKM9AwN9AwDgTHhxAABHAzSDEoMGsDwNmdwYPBgCGAIBdo7i+EAw0mGU0zSTI5MHkxeEKVMjFA9wgwogCkYANixoBo=</latexit> <latexit sha1_base64="Oa0xPtXVUZqDwpGLQAeyWNMf0Jk=">AAACmXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuYFGhUWCvElGSkliRqKtgqxKQVJSZXG9ZWR2lABWsVqgtq4wwUkNTFCygb6BmAgQImwxDKUHZweP9ij9gclYB8geUMMQwpDPkMyQylDLkMqQx5DCVAdg5DIkMxEEYzGDIYMBQAxWIZqoFiRUBWJlg+laGWgQuotxSoKhWoIhEomg0k04G8aKhoHpAPMrMYrDsZaEsOEBcBdSowqBpcNVhp8NnghMFqg5cGf3CaVQ02A+SWSiCdBNGbWhDP3yUR/J2grlwgXcKQgdCF180lDGkMFmC3ZgLdXgAWAfkiGaK/rGr652CrINVqNYNFBq+B7l9ocNPgMNAHeWVfkpcGpgbNZuACRoAhenBjMsKM9AwN9AwDgTHhxAABHAzSDEoMGsDwNmdwYPBgCGAIBdo7i+EAw0mGU0zSTI5MHkxeEKVMjFA9wgwogCkYANixoBo=</latexit> <latexit sha1_base64="v6WdAFhAQijKqZkSrsUrDZZNaYY=">AAACmXichVHLSiNBFD22r0x8RWcz4KYxKHETbougKILObGRWaiYqGg3dbSU26XQ33ZWgNvmB+YGBmdUMiIg/IbjR2c/CTxhcKrhx4U2nwRfqLarq1Kl7bp2qMjzbCiTRZZvS3tHZ1Z34kOzp7esfSA0OrQZuzTdF3nRt11839EDYliPy0pK2WPd8oVcNW6wZlS/N/bW68APLdb7JfU9sVfWyY5UsU5dMFVOTXmZvVi3IXSH1cXVOLZR83Qy1RriRicmGGnqNbVIf5RVTacpSFOpLoMUgjTiW3NQRCtiBCxM1VCHgQDK2oSPgtgkNBI+5LYTM+YysaF+ggSRra5wlOENntsJjmVebMevwulkziNQmn2Jz91mpYpT+0TFd0zmd0H+6e7VWGNVoetnn2WhphVcc+P4pd/uuqsqzxO6D6k3PEiVMR14t9u5FTPMWZktfP/hxnZtZGQ3H6A9dsf/fdElnfAOnfmMeLouVX0jyB2jPn/slWJ3IapTVlik9/zn+igSGMYIMv/cU5rGIJeT53J84xQX+KsPKgrKofG2lKm2x5iOehJK7B6+InDE=</latexit> p0(.; ✓) <latexit sha1_base64="uAUJ2Mb9zA7Ly6FPQCyZwgPfrWU=">AAACdnichVFNLwNBGH66vqo+2nKRSKQh9RFJ864L4SJcHBVFojS7a1ob293N7rRC4w/4Aw4iQUIjfoaLP+DgJ4gjCQcHb7dNBME7mZlnnnmfd56Z0V3L9CXRQ0hpam5pbQu3Rzo6u7qjsXjPqu+UPENkDMdyvHVd84Vl2iIjTWmJddcTWlG3xJq+O1/bXysLzzcde0Xuu2KzqBVsM28ammQqF4tX3MMtSoymZhJZuSOkNpaLDVGKgkj8BGoDDM0OZsfPACw6sSqy2IYDAyUUIWBDMragwee2ARUEl7lNVJjzGJnBvsAhIqwtcZbgDI3ZXR4LvNposDavazX9QG3wKRZ3j5UJJOmerumZ7uiGHun911qVoEbNyz7Pel0r3Fz0qG/59V9VkWeJnU/Vn54l8pgKvJrs3Q2Y2i2Mur58cPy8PL2UrAzTBT2x/3N6oFu+gV1+MS7TYukEEf4A9ftz/wSrEymVUmqaf2IO9QijH4MY5feexCwWsIgMn7uHU1yhGnpTBpSkMlJPVUINTS++hEIfpIaROA==</latexit> <latexit sha1_base64="w3lRR1wwNSwmVOx1hCd3eGQJBU8=">AAACdnichVFNLwNBGH66vuujxUUikYbURyTNuy6ES8PFEVUkSrO7hm5sdze700o1/oCzxEEcSBDxM1z8AYf+BHGshIODt9smguCdzMwzz7zPO8/M6K5l+pKoElKamlta29o7wp1d3T2RaG/fmu8UPEOkDcdyvA1d84Vl2iItTWmJDdcTWl63xLq+v1DbXy8Kzzcde1WWXLGV1/Zsc9c0NMlUNtpbdo+2KTaemItlZE5IbSIbHaEEBRH7CdQGGEkOZyZPKsnSkhO9QQY7cGCggDwEbEjGFjT43DahguAyt4Uycx4jM9gXOEKYtQXOEpyhMbvP4x6vNhuszetaTT9QG3yKxd1jZQxxeqRbqtID3dETvf9aqxzUqHkp8azXtcLNRo4HUq//qvI8S+Q+VX96ltjFTODVZO9uwNRuYdT1xcPTamp2JV4epUt6Zv8XVKF7voFdfDGulsXKGcL8Aer35/4J1qYSKiXUZf6JedSjHYMYxji/9zSSWMQS0nzuAc5xjZvQmzKkxJWxeqoSamj68SUU+gBq55K+</latexit> <latexit sha1_base64="w3lRR1wwNSwmVOx1hCd3eGQJBU8=">AAACdnichVFNLwNBGH66vuujxUUikYbURyTNuy6ES8PFEVUkSrO7hm5sdze700o1/oCzxEEcSBDxM1z8AYf+BHGshIODt9smguCdzMwzz7zPO8/M6K5l+pKoElKamlta29o7wp1d3T2RaG/fmu8UPEOkDcdyvA1d84Vl2iItTWmJDdcTWl63xLq+v1DbXy8Kzzcde1WWXLGV1/Zsc9c0NMlUNtpbdo+2KTaemItlZE5IbSIbHaEEBRH7CdQGGEkOZyZPKsnSkhO9QQY7cGCggDwEbEjGFjT43DahguAyt4Uycx4jM9gXOEKYtQXOEpyhMbvP4x6vNhuszetaTT9QG3yKxd1jZQxxeqRbqtID3dETvf9aqxzUqHkp8azXtcLNRo4HUq//qvI8S+Q+VX96ltjFTODVZO9uwNRuYdT1xcPTamp2JV4epUt6Zv8XVKF7voFdfDGulsXKGcL8Aer35/4J1qYSKiXUZf6JedSjHYMYxji/9zSSWMQS0nzuAc5xjZvQmzKkxJWxeqoSamj68SUU+gBq55K+</latexit> <latexit sha1_base64="fNaMbYxB+H4mwjCvQRiZp7000Gc=">AAACdnichVHLSsNAFD2N7/qquhEEKZb62JQbN4puim5camtV8FGSONZgmoRkWtHSH/AHXIgLBS3iZ7jxB1z0E8Slgi5ceJsGREW9w8ycOXPPnTMzumuZviSqR5SW1rb2js6uaHdPb19/bGBwzXdKniFyhmM53oau+cIybZGTprTEhusJrahbYl0/WGzsr5eF55uOvSqPXLFd1Aq2uWcammQqHxuouNUdik+m5uNbcl9IbSofS1CKgoj/BGoIEghj2YnVsIVdODBQQhECNiRjCxp8bptQQXCZ20aFOY+RGewLVBFlbYmzBGdozB7wWODVZsjavG7U9AO1wadY3D1WxpGkB7qhZ7qnW3qk919rVYIaDS9HPOtNrXDz/SfD2dd/VUWeJfY/VX96ltjDbODVZO9uwDRuYTT15ePT5+xcJlkZp0t6Yv8XVKc7voFdfjGuVkTmDFH+APX7c/8Ea9MplVLqCiXSC+FXdGIEY5jk955BGktYRo7PPcQ5rlGLvCmjSlKZaKYqkVAzhC+h0AfTnI+v</latexit> X x p0(x; ✓) <latexit sha1_base64="8k0CcF80RkwWl9z7U75UczfiOXc=">AAACe3ichVFdK0RBGH72+F4fuygpNycbIW3vUSJuRMmlr0VZtnOOwcn56pzZDcsf8AdcuKIk8SOUG3/AhZ8gl5Qbyrtnt4TwTjPzzDPv884zM4ZvW6EkeogpVdU1tXX1DfHGpuaWRLK1bSn08oEpMqZne8GKoYfCtlyRkZa0xYofCN0xbLFs7EyV9pcLIggtz12Ue75Yc/Qt19q0TF0ylUt2ZMO8k9tV/XVS+3bH1azcFlLvzyVTlKYo1J9Aq4DUxPRx+w2AWS95gSw24MFEHg4EXEjGNnSE3FahgeAzt4YicwEjK9oXOESctXnOEpyhM7vD4xavViusy+tSzTBSm3yKzT1gpYoeuqdLeqY7uqJHevu1VjGqUfKyx7NR1go/lzjqXHj9V+XwLLH9qfrTs8QmRiOvFnv3I6Z0C7OsL+wfPy+MzfcUe+mMntj/KT3QLd/ALbyY53Ni/gRx/gDt+3P/BEtDaY3S2hz/xCTKUY8udKOP33sEE5jBLDJ87gHOcIXr2LuSUgaUwXKqEqto2vEllOEPNlmTyw==</latexit> <latexit sha1_base64="H9PCfw1UrI5/SZpZDdMCzd8QFK8=">AAACe3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuIxRSX5sZXKBTEGShoVFgrxJRkpJYkasYLKBvoGYCBAibDEMpQdnCbLrxP6em9gHyB5QwxDCkM+QzJDKUMuQypDHkMJUB2DkMiQzEQRjMYMhgwFADFYhmqgWJFQFYmWD6VoZaBC6i3FKgqFagiESiaDSTTgbxoqGgekA8ysxisOxloSw4QFwF1KjCoGlw1WGnw2eCEwWqDlwZ/cJpVDTYD5JZKIJ0E0ZtaEM/fJRH8naCuXCBdwpCB0IXXzSUMaQwWYLdmAt1eABYB+SIZor+savrnYKsg1Wo1g0UGr4HuX2hw0+Aw0Ad5ZV+SlwamBs1m4AJGgCF6cGMywoz0DA30DAOBMeHEAAEcDNIMSgwawPA2Z3Bg8GAIYAgF2lvDsIhhNcMaxr9MykxaTDoQpUyMUD3CDCiAyRQApPyVrg==</latexit> <latexit sha1_base64="H9PCfw1UrI5/SZpZDdMCzd8QFK8=">AAACe3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuIxRSX5sZXKBTEGShoVFgrxJRkpJYkasYLKBvoGYCBAibDEMpQdnCbLrxP6em9gHyB5QwxDCkM+QzJDKUMuQypDHkMJUB2DkMiQzEQRjMYMhgwFADFYhmqgWJFQFYmWD6VoZaBC6i3FKgqFagiESiaDSTTgbxoqGgekA8ysxisOxloSw4QFwF1KjCoGlw1WGnw2eCEwWqDlwZ/cJpVDTYD5JZKIJ0E0ZtaEM/fJRH8naCuXCBdwpCB0IXXzSUMaQwWYLdmAt1eABYB+SIZor+savrnYKsg1Wo1g0UGr4HuX2hw0+Aw0Ad5ZV+SlwamBs1m4AJGgCF6cGMywoz0DA30DAOBMeHEAAEcDNIMSgwawPA2Z3Bg8GAIYAgF2lvDsIhhNcMaxr9MykxaTDoQpUyMUD3CDCiAyRQApPyVrg==</latexit> <latexit sha1_base64="2XjVx2/Q0RR5744YaXto0PhCGiY=">AAACe3ichVHLLgRBFD3T3uM1SERi0zEhiExuS4SwETaWXoPEMOluNXT0K901Eww/4AcsrEhEhL+w8QMWPkEsSWxI3OnpRBDcSlWdOnXPrVNVhm9boSR6SCg1tXX1DY1NyeaW1rb2VEfnSugVA1NkTc/2gjVDD4VtuSIrLWmLNT8QumPYYtXYna3sr5ZEEFqeuyz3fbHh6NuuVbBMXTKVT3XnwqKT31P9TVIH96bUnNwRUh/Kp9KUoSjUn0CLQRpxzHupS+SwBQ8minAg4EIytqEj5LYODQSfuQ2UmQsYWdG+wBGSrC1yluAMndldHrd5tR6zLq8rNcNIbfIpNveAlSr66Z6u6Jnu6Joe6e3XWuWoRsXLPs9GVSv8fPtxz9LrvyqHZ4mdT9WfniUKmIi8Wuzdj5jKLcyqvnRw8rw0udhfHqBzemL/Z/RAt3wDt/RiXiyIxVMk+QO078/9E6yMZjTKaAuUnp6Jv6IRvejDIL/3OKYxh3lk+dxDnOMaN4l3Ja0MKyPVVCURa7rwJZSxD6hKkds=</latexit> Z p0(x; ✓)dx <latexit sha1_base64="WNLv6W7OFBixBnjBW4pOz/mIEdo=">AAAChXichVG7ThtBFD1sQjCGBAcaJJoVCARFrLtRICgUWNBQAsbYEgZrdxnwyPvS7tiysWgp+AEKKpCiCCVt+AEafoDCn4BSEomGguu1JURQkjuamTNn7rlzZsYKHBkpolaP9up175u+RH9yYPDtu6HU++HNyK+GtsjZvuOHBcuMhCM9kVNSOaIQhMJ0LUfkrcpyez9fE2EkfW9DNQKx7Zr7ntyTtqmYKqX0ovSUHuyQPl1f0IuqLJQ5oxddU5VDt7l7WC+lJihNcegvgdEFE5nF9GwGwKqf+oYiduHDRhUuBDwoxg5MRNy2YIAQMLeNJnMhIxnvCxwiydoqZwnOMJmt8LjPq60u6/G6XTOK1Taf4nAPWaljkm7ogu7omr7TLT38tVYzrtH20uDZ6mhFUBo6Hs3e/1fl8qxQflL907PCHuZjr5K9BzHTvoXd0dcOTu6yX9Ynm1N0Tr/Y/xm16Ipv4NV+21/XxPopkvwBxp/P/RJsfkwblDbW+CeW0IkExjCOaX7vz8hgBavI8blH+IGfuNT6tA/aJ22uk6r1dDUjeBba4iNTfZZa</latexit> <latexit sha1_base64="BTMM6AXpgMIWmsAOwd8cGDdlNys=">AAAChXichVFNL2RBFD0eBs2MxkZi80KIWUznPvEVFi1sLH01Em067z1FV7yvvFfdQbO1sLOysCIRkZktf8DGH7DwE0SsSGws3H7diRiZmVupqlOn7rl1qsoKHBkporsarbau/ktDY1OiueXrt9ZkW/ti5BdCW2Rs3/HDZcuMhCM9kVFSOWI5CIXpWo5YsjanyvtLRRFG0vcW1HYgVl1zw5Pr0jYVU7mknpWe0oOfpPdvjetZlRfK/K5nXVPlQ7e0treVS/ZQiuLQPwOjCnom0qmh9O7j4YyfPEcWa/BhowAXAh4UYwcmIm4rMEAImFtFibmQkYz3BfaQYG2BswRnmMxu8rjBq5Uq6/G6XDOK1Taf4nAPWamjl27pgp7ohn7RPb3+tVYprlH2ss2zVdGKINd60Dn/8l+Vy7NC/l31T88K6xiNvUr2HsRM+RZ2RV/cOXqaH5vrLfXRKT2w/xO6o2u+gVd8ts9mxdwxEvwBxp/P/RksDqQMShmz/BOTqEQjutCNfn7vEUxgGjPI8Ln7+I1LXGkN2g9tUBuupGo1VU0HPoSWfgPq1ZhT</latexit> <latexit sha1_base64="BTMM6AXpgMIWmsAOwd8cGDdlNys=">AAAChXichVFNL2RBFD0eBs2MxkZi80KIWUznPvEVFi1sLH01Em067z1FV7yvvFfdQbO1sLOysCIRkZktf8DGH7DwE0SsSGws3H7diRiZmVupqlOn7rl1qsoKHBkporsarbau/ktDY1OiueXrt9ZkW/ti5BdCW2Rs3/HDZcuMhCM9kVFSOWI5CIXpWo5YsjanyvtLRRFG0vcW1HYgVl1zw5Pr0jYVU7mknpWe0oOfpPdvjetZlRfK/K5nXVPlQ7e0treVS/ZQiuLQPwOjCnom0qmh9O7j4YyfPEcWa/BhowAXAh4UYwcmIm4rMEAImFtFibmQkYz3BfaQYG2BswRnmMxu8rjBq5Uq6/G6XDOK1Taf4nAPWamjl27pgp7ohn7RPb3+tVYprlH2ss2zVdGKINd60Dn/8l+Vy7NC/l31T88K6xiNvUr2HsRM+RZ2RV/cOXqaH5vrLfXRKT2w/xO6o2u+gVd8ts9mxdwxEvwBxp/P/RksDqQMShmz/BOTqEQjutCNfn7vEUxgGjPI8Ln7+I1LXGkN2g9tUBuupGo1VU0HPoSWfgPq1ZhT</latexit> <latexit sha1_base64="wYtk8KKaebSYh98G52k+qXbQykU=">AAAChXichVG7ThtBFD1sABuHh5M0kWhWWESmwLqLEERJAQoNJY/YIGFj7S4DHrEv7Y4tG4s2RX4gBRVIKIqghR+g4QdS+BOilERKkyLX65UQIOCOZubMmXvunJmxAkdGiqjTp73oHxhMpYcyL4dHRseyr16XIr8e2qJo+44fblpmJBzpiaKSyhGbQShM13LEhrW/1N3faIgwkr73WbUCUXHNPU/uSttUTFWzell6Sg+2Sc83P+plVRPKnNLLrqlqodveOWxWszkqUBz6Q2AkIIckVvzsd5SxAx826nAh4EExdmAi4rYFA4SAuQrazIWMZLwvcIgMa+ucJTjDZHafxz1ebSWsx+tuzShW23yKwz1kpY5J+kk/6Iau6Yx+0b9Ha7XjGl0vLZ6tnlYE1bGvb9f/PqtyeVao3aqe9Kywi/exV8neg5jp3sLu6RsH327WP6xNtt/RCf1m/8fUoSu+gdf4Y5+uirUjZPgDjPvP/RCUZgoGFYxVyi1+Sr4ijXFMIM/vPY9FLGMFRT73C85xgUstpU1rs9pcL1XrSzRvcCe0hf8+6ZU4</latexit> Unnormalized model: s.t. Z p0(x; ✓)dx 6= 1 <latexit sha1_base64="Xen8d6aBoVjqw8xtsIG75vP5hoM=">AAACinichVG7ThtBFD1sCBDzMqGJRLPCcgSNdRch8WpQSEHJIwYklli7yxiP2Be7Ywti8QPwASmoQEIIUdFCSZMfoOATECVINBS5Xq8UBQTc0cycOXPPnTMzdujKWBHdtGgfWj+2tXd8ynR2dff0Zvs+L8VBNXJE0QncIFqxrVi40hdFJZUrVsJIWJ7timV7c6axv1wTUSwD/4faCcWaZ234siwdSzFVyuZN6Ss9/ElD21OmqghlDeumZ6lK5NXXd7d10xdbulHK5qhASegvgZGCHNKYC7InMLGOAA6q8CDgQzF2YSHmtgoDhJC5NdSZixjJZF9gFxnWVjlLcIbF7CaPG7xaTVmf142acaJ2+BSXe8RKHXm6plO6pz90Rrf09GqtelKj4WWHZ7upFWGpd+/L4uO7Ko9nhco/1ZueFcoYT7xK9h4mTOMWTlNf+/X7fnFyIV//Skd0x/4P6Yau+AZ+7cE5nhcLB8jwBxjPn/slWBopGFQw5kdz09/Sr+jAAAYxxO89hmnMYg5FPncf57jApdaljWgT2lQzVWtJNf34L7TvfwHnhpc/</latexit> <latexit sha1_base64="Xen8d6aBoVjqw8xtsIG75vP5hoM=">AAACinichVG7ThtBFD1sCBDzMqGJRLPCcgSNdRch8WpQSEHJIwYklli7yxiP2Be7Ywti8QPwASmoQEIIUdFCSZMfoOATECVINBS5Xq8UBQTc0cycOXPPnTMzdujKWBHdtGgfWj+2tXd8ynR2dff0Zvs+L8VBNXJE0QncIFqxrVi40hdFJZUrVsJIWJ7timV7c6axv1wTUSwD/4faCcWaZ234siwdSzFVyuZN6Ss9/ElD21OmqghlDeumZ6lK5NXXd7d10xdbulHK5qhASegvgZGCHNKYC7InMLGOAA6q8CDgQzF2YSHmtgoDhJC5NdSZixjJZF9gFxnWVjlLcIbF7CaPG7xaTVmf142acaJ2+BSXe8RKHXm6plO6pz90Rrf09GqtelKj4WWHZ7upFWGpd+/L4uO7Ko9nhco/1ZueFcoYT7xK9h4mTOMWTlNf+/X7fnFyIV//Skd0x/4P6Yau+AZ+7cE5nhcLB8jwBxjPn/slWBopGFQw5kdz09/Sr+jAAAYxxO89hmnMYg5FPncf57jApdaljWgT2lQzVWtJNf34L7TvfwHnhpc/</latexit> <latexit sha1_base64="Xen8d6aBoVjqw8xtsIG75vP5hoM=">AAACinichVG7ThtBFD1sCBDzMqGJRLPCcgSNdRch8WpQSEHJIwYklli7yxiP2Be7Ywti8QPwASmoQEIIUdFCSZMfoOATECVINBS5Xq8UBQTc0cycOXPPnTMzdujKWBHdtGgfWj+2tXd8ynR2dff0Zvs+L8VBNXJE0QncIFqxrVi40hdFJZUrVsJIWJ7timV7c6axv1wTUSwD/4faCcWaZ234siwdSzFVyuZN6Ss9/ElD21OmqghlDeumZ6lK5NXXd7d10xdbulHK5qhASegvgZGCHNKYC7InMLGOAA6q8CDgQzF2YSHmtgoDhJC5NdSZixjJZF9gFxnWVjlLcIbF7CaPG7xaTVmf142acaJ2+BSXe8RKHXm6plO6pz90Rrf09GqtelKj4WWHZ7upFWGpd+/L4uO7Ko9nhco/1ZueFcoYT7xK9h4mTOMWTlNf+/X7fnFyIV//Skd0x/4P6Yau+AZ+7cE5nhcLB8jwBxjPn/slWBopGFQw5kdz09/Sr+jAAAYxxO89hmnMYg5FPncf57jApdaljWgT2lQzVWtJNf34L7TvfwHnhpc/</latexit> <latexit sha1_base64="Xen8d6aBoVjqw8xtsIG75vP5hoM=">AAACinichVG7ThtBFD1sCBDzMqGJRLPCcgSNdRch8WpQSEHJIwYklli7yxiP2Be7Ywti8QPwASmoQEIIUdFCSZMfoOATECVINBS5Xq8UBQTc0cycOXPPnTMzdujKWBHdtGgfWj+2tXd8ynR2dff0Zvs+L8VBNXJE0QncIFqxrVi40hdFJZUrVsJIWJ7timV7c6axv1wTUSwD/4faCcWaZ234siwdSzFVyuZN6Ss9/ElD21OmqghlDeumZ6lK5NXXd7d10xdbulHK5qhASegvgZGCHNKYC7InMLGOAA6q8CDgQzF2YSHmtgoDhJC5NdSZixjJZF9gFxnWVjlLcIbF7CaPG7xaTVmf142acaJ2+BSXe8RKHXm6plO6pz90Rrf09GqtelKj4WWHZ7upFWGpd+/L4uO7Ko9nhco/1ZueFcoYT7xK9h4mTOMWTlNf+/X7fnFyIV//Skd0x/4P6Yau+AZ+7cE5nhcLB8jwBxjPn/slWBopGFQw5kdz09/Sr+jAAAYxxO89hmnMYg5FPncf57jApdaljWgT2lQzVWtJNf34L7TvfwHnhpc/</latexit>
  4. Intuition of Noise Contrastive Estimation (NCE) Solving binary classification problem!

    4 %BUBTBNQMFT /PJTFTBNQMFT data? or noise? Logistic Regression It can be considered as a fixed generator and a trainable discriminator in the GAN game.
  5. Overview of NCE 1. Replace an expensive partition function with

    
 a learnable parameter in a trained model 2. Sample noise data from a noise distribution 3. Classify between observed data and noise data 5
  6. Overview of NCE 1. Replace an expensive partition function with

    
 a learnable parameter in a trained model 2. Sample noise data from a noise distribution 3. Classify between observed data and noise data 6
  7. Replace a partition function with a learnable parameter 7 is

    a learnable parameter instead of the partition function c <latexit sha1_base64="dqnNk2e//D8T2u6qA6gageMEH+k=">AAACZHichVFNS8NAFJzGr1qrrRZBEEQsiqfy4kURxKIXj9ZaLdRSkrhqaJqEJC3UInjWq+LBk4KI+DM86B/w0D8giEcFLx58TQuiRX3L7s7Ovnk7u6vahu56RPWA1NHZ1d0T7A31hfsHItHBoQ3XKjuayGiWYTlZVXGFoZsi4+meIbK2I5SSaohNtbjc2N+sCMfVLXPdq9oiX1J2TX1H1xSPqZRWiMYpQX6MtwO5BeKL96GFQwCrVvQaW9iGBQ1llCBgwmNsQIHLLQcZBJu5PGrMOYx0f1/gACHWljlLcIbCbJHHXV7lWqzJ60ZN11drfIrB3WHlOCbpkW7olR7olp7p49daNb9Gw0uVZ7WpFXYhcjSSfv9XVeLZw96X6k/PHnYw53vV2bvtM41baE19Zf/sNT2/Nlmbokt6Yf8XVKc7voFZedOuUmLtHCH+APnnc7eDjZmETAk5RfHkEpoRxCgmMM3vPYskVrCKDJ8rcIwTnAaepLAUk4abqVKgpYnhW0hjnxmui6w=</latexit> <latexit sha1_base64="LQr8q1wW7eSqPsXuz5Dptsecmko=">AAACZHichVFdKwRRGH52fK2x7CKllGQjV9s7bkjJxo1Li0WhbWacXZP5amZ2i80f4JZcuEKS/AwX/AHJH1BySblx4Z3ZLSG8p3POc57zPu95zjmaaxp+QPQQkxoam5pb4q1yW6K9I5nq7FrynbKni7zumI63oqm+MA1b5AMjMMWK6wnV0kyxrG3NhPvLFeH5hmMvBtuuWLfUkm0UDV0NmMrphVSaMhTFwE+g1EF66kaedM/u5TkndYE1bMCBjjIsCNgIGJtQ4XNbhQKCy9w6qsx5jIxoX2AXMmvLnCU4Q2V2i8cSr1brrM3rsKYfqXU+xeTusXIAQ3RHl/RCt3RFT/T+a61qVCP0ss2zVtMKt5Dc6114+1dl8Rxg81P1p+cARYxHXg327kZMeAu9pq/sHL0sTMwPVYfplJ7Z/wk90DXfwK686uc5MX8MmT9A+f7cP8HSaEahjJKjdHYatYijD4MY4fceQxazmEOezxXYxwEOY49SQuqWemqpUqyu6caXkPo/AL9tjSA=</latexit> <latexit sha1_base64="LQr8q1wW7eSqPsXuz5Dptsecmko=">AAACZHichVFdKwRRGH52fK2x7CKllGQjV9s7bkjJxo1Li0WhbWacXZP5amZ2i80f4JZcuEKS/AwX/AHJH1BySblx4Z3ZLSG8p3POc57zPu95zjmaaxp+QPQQkxoam5pb4q1yW6K9I5nq7FrynbKni7zumI63oqm+MA1b5AMjMMWK6wnV0kyxrG3NhPvLFeH5hmMvBtuuWLfUkm0UDV0NmMrphVSaMhTFwE+g1EF66kaedM/u5TkndYE1bMCBjjIsCNgIGJtQ4XNbhQKCy9w6qsx5jIxoX2AXMmvLnCU4Q2V2i8cSr1brrM3rsKYfqXU+xeTusXIAQ3RHl/RCt3RFT/T+a61qVCP0ss2zVtMKt5Dc6114+1dl8Rxg81P1p+cARYxHXg327kZMeAu9pq/sHL0sTMwPVYfplJ7Z/wk90DXfwK686uc5MX8MmT9A+f7cP8HSaEahjJKjdHYatYijD4MY4fceQxazmEOezxXYxwEOY49SQuqWemqpUqyu6caXkPo/AL9tjSA=</latexit> <latexit sha1_base64="dxmZ8Hqd0XIx3n6Mwtt8nkpiKvQ=">AAACZHichVG7SgNBFD1Z3/GRqAiCIMEQsQp3bRSroI2lMeYBUWR3neiSfbG7CcTgD2irWFgpiIifYeMPWOQHBLGMYGPhzWZBVNQ7zMyZM/fcOTOjOobu+UStiNTT29c/MDgUHR4ZHYvFxycKnl1zNZHXbMN2S6riCUO3RN7XfUOUHFcopmqIolpd6+wX68L1dNva8huO2DGVfUuv6JriM5XVduNJSlMQiZ9ADkESYWzY8RtsYw82NNRgQsCCz9iAAo9bGTIIDnM7aDLnMtKDfYEjRFlb4yzBGQqzVR73eVUOWYvXnZpeoNb4FIO7y8oEUvRIt9SmB7qjZ3r/tVYzqNHx0uBZ7WqFsxs7ns69/asyefZx8Kn607OPCpYDrzp7dwKmcwutq68fnrdzK5up5jxd0Qv7v6QW3fMNrPqrdp0VmxeI8gfI35/7JygspmVKy1lKZlbDrxjEDOawwO+9hAzWsYE8nytwglOcRZ6kEWlSmuqmSpFQM4kvIc1+AMsMid8=</latexit> Ex: 1D Gaussian model Original: NCE: pm(x; µ, ) = 1 p 2⇡ 2 exp (x µ)2 2 2 <latexit sha1_base64="VSlY5Qjqb/hIP4PkB8BlzrqPZx0=">AAAC2nichZFNa9RAGMf/iW91q3bVi+BlsFR2wS6TvSiKsOjFY1/cttCpaxJn16F5GZPJsnXYizf1KnjwpCAifgwvfgA9FD+BeKzgxYNPsitii/qEZJ75z//3zDOZQEcqN5zvOu6hw0eOHps5Xps9cfLUXP30mbU8LbJQdsM0SrONwM9lpBLZNcpEckNn0o+DSK4H2zfL9fWhzHKVJrfNjpZbsT9IVF+FviGpVy90L26M2DUm4uISE7kaxH6TsetW9DM/ZNYbW5E/yAyzbaHV1MDu2PaYggk50kwEatCwi7+IxmiRarFm5SFqH0HmZq1Xn+ctXgU7mHjTZL7T+ajvAlhK628gcA8pQhSIIZHAUB7BR07PJjxwaNK2YEnLKFPVusQYNWILckly+KRu03dAs82pmtC8rJlXdEi7RPRmRDIs8E/8Ld/jH/g7/oX/+GstW9Uoe9mhMZiwUvfmnpxb/f5fKqbR4P5v6p89G/RxpepVUe+6UspThBN++PD53urVlQV7kb/iX6n/l3yXv6cTJMNv4etlufIC5QV4+3/3wWSt3fJ4y1umm7iBSczgPC6gQf/7Mjq4hSV0ad/PjuPUnFlXuI/cx+7TidV1psxZ/BHus5+6BbUk</latexit> <latexit sha1_base64="vc6E7/f7WSYPE/Kug47h1XxApFk=">AAAC2nichZG/axRBFMffrr+Si5rTNAGbwZBwB+Z4e41BEY7YWOaHlwQy8dhd5y5D9sdkd/a4ZLgmXbQVLKwURMQ/w8ZatAj5C4JlhDQp8nbvRExQ37I7b77z/bx5s+OpQKYa8cCyL12+cvXayGhp7PqNm+PlW7dX0jhLfNH04yBO1jw3FYGMRFNLHYg1lQg39AKx6m09ztdXuyJJZRw91TtKbIRuJ5Jt6buapFY5U62w0mMPGQ+ze4ynshO6VcYeGd5OXJ8Zp294up1oZupcyaGBPTP1PgXjoqcY92SnYmZ/EZXeLNVi1cJD1DmCzNVSqzyFNSyCXUycYTLVaHxTfD/8uhCXPwCH5xCDDxmEICACTXkALqT0rIMDCIq0DTCkJZTJYl1AH0rEZuQS5HBJ3aJvh2brQzWieV4zLWifdgnoTYhkMI3f8SMe4xf8hEd4+tdapqiR97JDozdghWqNv5hcPvkvFdKoYfM39c+eNbRhruhVUu+qUPJT+AO+u/v6ePnB0rSZwXf4g/p/iwf4mU4QdX/67xfF0hvIL8A5/7svJiv1moM1Z5FuYh4GMQJ34C5U6H/fhwY8gQVo0r6HlmWVrDGb23v2vv1yYLWtITMBf4T96gzwqbbo</latexit> <latexit sha1_base64="vc6E7/f7WSYPE/Kug47h1XxApFk=">AAAC2nichZG/axRBFMffrr+Si5rTNAGbwZBwB+Z4e41BEY7YWOaHlwQy8dhd5y5D9sdkd/a4ZLgmXbQVLKwURMQ/w8ZatAj5C4JlhDQp8nbvRExQ37I7b77z/bx5s+OpQKYa8cCyL12+cvXayGhp7PqNm+PlW7dX0jhLfNH04yBO1jw3FYGMRFNLHYg1lQg39AKx6m09ztdXuyJJZRw91TtKbIRuJ5Jt6buapFY5U62w0mMPGQ+ze4ynshO6VcYeGd5OXJ8Zp294up1oZupcyaGBPTP1PgXjoqcY92SnYmZ/EZXeLNVi1cJD1DmCzNVSqzyFNSyCXUycYTLVaHxTfD/8uhCXPwCH5xCDDxmEICACTXkALqT0rIMDCIq0DTCkJZTJYl1AH0rEZuQS5HBJ3aJvh2brQzWieV4zLWifdgnoTYhkMI3f8SMe4xf8hEd4+tdapqiR97JDozdghWqNv5hcPvkvFdKoYfM39c+eNbRhruhVUu+qUPJT+AO+u/v6ePnB0rSZwXf4g/p/iwf4mU4QdX/67xfF0hvIL8A5/7svJiv1moM1Z5FuYh4GMQJ34C5U6H/fhwY8gQVo0r6HlmWVrDGb23v2vv1yYLWtITMBf4T96gzwqbbo</latexit> <latexit sha1_base64="nh/59HsTMZFqmDxb3h0CAEJndIA=">AAAC2nichZHPa9RAFMdfYtW6Vbu1F8HL4FLZBbu87KVFKRS9eOwPty106pLE2XVofkyTybLtsJfetNeCB08KIsU/w4v/gIfiXyAeK/TSgy/ZiNiivpDMm+98P2/eZDwVyFQjHlv2pbHLV66OX6tMXL9xc7I6dWstjbPEF20/DuJkw3NTEchItLXUgdhQiXBDLxDr3vbjfH29L5JUxtFTvavEVuj2ItmVvqtJ6lQz1QnrA/aQ8TC7z3gqe6HbYGzB8G7i+sw4Q8PTnUQz0+JKlgb2zLSGFIyLgWLck726mf1F1AezVIs1Cg9R5wgyNyqdag2bWAS7mDhlUoMyluLqB+DwHGLwIYMQBESgKQ/AhZSeTXAAQZG2BYa0hDJZrAsYQoXYjFyCHC6p2/Tt0WyzVCOa5zXTgvZpl4DehEgGM/gFj/AEP+NH/IZnf61lihp5L7s0eiNWqM7kq9urp/+lQho1vPhN/bNnDV2YL3qV1LsqlPwU/ojv770+WX2wMmPu4Tv8Tv2/xWP8RCeI+j/898ti5Q3kF+Cc/90Xk7VW08Gms4y1xUflVYzDHbgLdfrfc7AIT2AJ2rTvV8uyKtaEze19+6V9MLLaVslMwx9hH/4EzCGzAA==</latexit> pm(x; c, µ, ) = 1 c exp (x µ)2 2 2 <latexit sha1_base64="8TaLo4nCxuio5CW5jRO+26vEj8M=">AAACxnichZFLaxRBEMf/GV9xfWTVi5BLY4jsgll69mJQhKCXHPNwk0AmLj1t79pkXsxj3dgMePYLePCkEELwY3jQoxcPOXkW8RQxlxxSM7MhaFB7mO6qf9WvurrbjTydpJzvjVlnzp47f2H8Yu3S5StXJ+rXrq8kYRZL1ZGhF8ZrrkiUpwPVSXXqqbUoVsJ3PbXqbj4q4qsDFSc6DB6nW5Ha8EU/0D0tRUpSty6irt8YsvtM3mGOn9GU6L4vmow9ME4vFpIZOzcyz5mjhhFzXN1vmJnjUGM4QxBrPjHtPDftimWlVxCU3Kx161O8xcvBThv2yJiam/36mQFYCOs7cPAUISQy+FAIkJLtQSChbx02OCLSNmBIi8nSZVwhR43YjLIUZQhSN2nuk7c+UgPyi5pJSUvaxaM/JpJhmn/hu3yff+Lv+Td++NdapqxR9LJFq1uxKupOvLq5fPBfyqc1xbMT6p89p+hhtuxVU+9RqRSnkBU/ePF6f/ne0rS5zd/x79T/W77HP9AJgsFPub2olt6geAD7z+s+bay0WzZv2Yv0Eg9RjXFM4hYadN93MYd5LKBD+37ED/zCgTVvBVZmPa9SrbERcwO/DevlEdPWr7Q=</latexit> <latexit sha1_base64="rt302ArbxwqnOirDpZtry4NzGNg=">AAACxnichZFLaxRBEMf/GV9xfWTVg4qXxhDZBbP07MWgBEK85JiHmwQycelpe9cm82Ie6ybNgGe/QA6eFETEj+FBj148xItXRTxFzMWDNTMrokHtYbqr/lW/6upuN/J0knK+N2YdOXrs+Inxk7VTp8+cnaifO7+ahFksVUeGXhivuyJRng5UJ9Wpp9ajWAnf9dSau3W7iK8NVJzoMLiTbkdq0xf9QPe0FClJ3bqIun5jyG4xeZ05fkZTovu+aDI2a5xeLCQzdm5knjNHDSPmuLrfMNM/Q43hNEGsede089y0K5aVXkFQcrPWrU/yFi8HO2zYI2NybubD24vvP15eDOvP4eAeQkhk8KEQICXbg0BC3wZscESkbcKQFpOly7hCjhqxGWUpyhCkbtHcJ29jpAbkFzWTkpa0i0d/TCTDFH/HX/B9/oa/5J/497/WMmWNopdtWt2KVVF34tGllYP/Uj6tKe7/ov7Zc4oeZspeNfUelUpxClnxg53d/ZWby1PmGn/KP1P/T/gef0UnCAZf5bMltfwYxQPYf173YWO13bJ5y16il5hHNcZxBVfRoPu+gTksYBEd2vc1vuAbDqwFK7Ay60GVao2NmAv4bVgPfwAlBbGG</latexit> <latexit sha1_base64="rt302ArbxwqnOirDpZtry4NzGNg=">AAACxnichZFLaxRBEMf/GV9xfWTVg4qXxhDZBbP07MWgBEK85JiHmwQycelpe9cm82Ie6ybNgGe/QA6eFETEj+FBj148xItXRTxFzMWDNTMrokHtYbqr/lW/6upuN/J0knK+N2YdOXrs+Inxk7VTp8+cnaifO7+ahFksVUeGXhivuyJRng5UJ9Wpp9ajWAnf9dSau3W7iK8NVJzoMLiTbkdq0xf9QPe0FClJ3bqIun5jyG4xeZ05fkZTovu+aDI2a5xeLCQzdm5knjNHDSPmuLrfMNM/Q43hNEGsede089y0K5aVXkFQcrPWrU/yFi8HO2zYI2NybubD24vvP15eDOvP4eAeQkhk8KEQICXbg0BC3wZscESkbcKQFpOly7hCjhqxGWUpyhCkbtHcJ29jpAbkFzWTkpa0i0d/TCTDFH/HX/B9/oa/5J/497/WMmWNopdtWt2KVVF34tGllYP/Uj6tKe7/ov7Zc4oeZspeNfUelUpxClnxg53d/ZWby1PmGn/KP1P/T/gef0UnCAZf5bMltfwYxQPYf173YWO13bJ5y16il5hHNcZxBVfRoPu+gTksYBEd2vc1vuAbDqwFK7Ay60GVao2NmAv4bVgPfwAlBbGG</latexit> <latexit sha1_base64="B0ExEql2Vc5wEZ8QFOvDSkQ05K8=">AAACxnichZFLaxRBEMf/GV9xfWTVi+ClcYnsgllq9qIoQtBLjnm4SSATl5m2d9NkXsxj3dgMePYLePCkICJ+DA/6BTzkI4h4ipiLB2tmR0SD2sN0V/2rftXV3V7s6zQj2p+xjh0/cfLU7OnGmbPnzs81L1xcT6M8kaovIz9KNj03Vb4OVT/Tma8240S5geerDW/3XhnfGKsk1VF4P9uL1XbgjkI91NLNWBo03XgQtCfitpDXhRPkPKV6FLgdIe4YZ5i4Uhi7MLIohKMmsXA8PWqbhZ+h9mSBIdF5YHpFYXpTVlReSXBypzFotqhL1RBHDbs2WqjHctR8DQcPEUEiRwCFEBnbPlyk/G3BBiFmbRuGtYQtXcUVCjSYzTlLcYbL6i7PI/a2ajVkv6yZVrTkXXz+EyYF5ukjvaED+kBv6RN9/2stU9Uoe9nj1ZuyKh7MPb28dvhfKuA1w84v6p89ZxjiZtWr5t7jSilPIaf8+PGzg7Vbq/PmGr2kz9z/C9qnd3yCcPxVvlpRq89RPoD953UfNdZ7XZu69gq1Fu/WTzGLK7iKNt/3DSxiCcvo877v8QXfcGgtWaGVW4+mqdZMzVzCb8N68gOlmK1t</latexit> pm(x; ✓) = 1 Z(✓) p0 m (x; ✓) <latexit sha1_base64="KZAh7y0V0qe3/C6DUlQ3EjB9tIY=">AAAC3XichVFLLwRBEK4Z7/VaXCQuHZsVLpsakRAiES6OXouwTGZaLxPzykzvBps9uoi4OjiRiIif4eIPOLi4iyOJi4Pa2Qlhg5pM99df1Vf9dbfp21YoER8Uta6+obGpuSXR2tbe0Zns6l4OvULARZZ7thesmkYobMsVWWlJW6z6gTAc0xYr5u5MJb9SFEFoee6S3PfFhmNsu1be4oYkSk/u+brDBtkem2A5uSOkwYbYJMvlA4OzEtPKNKxR/jNXZj7bJBLLtbo0+y7kf1XryRRmMApWC7QYpCCOOS95BTnYAg84FMABAS5IwjYYENK3Dhog+MRtQIm4gJAV5QWUIUHaAlUJqjCI3aVxm1brMevSutIzjNScdrHpD0jJII33eI0veIc3+ITvv/YqRT0qXvZpNqta4eudR72Lb/+qHJol7Hyp/vQsIQ9jkVeLvPsRUzkFr+qLB6cvi+ML6dIAXuAz+T/HB7ylE7jFV345LxbOIEEPoP287lqwPJzRMKPNj6SmpuOnaIY+6IdBuu9RmIJZmIMs7fuo1CvtSoeqq4fqsXpSLVWVWNMD30I9/QA8fKyx</latexit> <latexit sha1_base64="KZAh7y0V0qe3/C6DUlQ3EjB9tIY=">AAAC3XichVFLLwRBEK4Z7/VaXCQuHZsVLpsakRAiES6OXouwTGZaLxPzykzvBps9uoi4OjiRiIif4eIPOLi4iyOJi4Pa2Qlhg5pM99df1Vf9dbfp21YoER8Uta6+obGpuSXR2tbe0Zns6l4OvULARZZ7thesmkYobMsVWWlJW6z6gTAc0xYr5u5MJb9SFEFoee6S3PfFhmNsu1be4oYkSk/u+brDBtkem2A5uSOkwYbYJMvlA4OzEtPKNKxR/jNXZj7bJBLLtbo0+y7kf1XryRRmMApWC7QYpCCOOS95BTnYAg84FMABAS5IwjYYENK3Dhog+MRtQIm4gJAV5QWUIUHaAlUJqjCI3aVxm1brMevSutIzjNScdrHpD0jJII33eI0veIc3+ITvv/YqRT0qXvZpNqta4eudR72Lb/+qHJol7Hyp/vQsIQ9jkVeLvPsRUzkFr+qLB6cvi+ML6dIAXuAz+T/HB7ylE7jFV345LxbOIEEPoP287lqwPJzRMKPNj6SmpuOnaIY+6IdBuu9RmIJZmIMs7fuo1CvtSoeqq4fqsXpSLVWVWNMD30I9/QA8fKyx</latexit> <latexit sha1_base64="KZAh7y0V0qe3/C6DUlQ3EjB9tIY=">AAAC3XichVFLLwRBEK4Z7/VaXCQuHZsVLpsakRAiES6OXouwTGZaLxPzykzvBps9uoi4OjiRiIif4eIPOLi4iyOJi4Pa2Qlhg5pM99df1Vf9dbfp21YoER8Uta6+obGpuSXR2tbe0Zns6l4OvULARZZ7thesmkYobMsVWWlJW6z6gTAc0xYr5u5MJb9SFEFoee6S3PfFhmNsu1be4oYkSk/u+brDBtkem2A5uSOkwYbYJMvlA4OzEtPKNKxR/jNXZj7bJBLLtbo0+y7kf1XryRRmMApWC7QYpCCOOS95BTnYAg84FMABAS5IwjYYENK3Dhog+MRtQIm4gJAV5QWUIUHaAlUJqjCI3aVxm1brMevSutIzjNScdrHpD0jJII33eI0veIc3+ITvv/YqRT0qXvZpNqta4eudR72Lb/+qHJol7Hyp/vQsIQ9jkVeLvPsRUzkFr+qLB6cvi+ML6dIAXuAz+T/HB7ylE7jFV345LxbOIEEPoP287lqwPJzRMKPNj6SmpuOnaIY+6IdBuu9RmIJZmIMs7fuo1CvtSoeqq4fqsXpSLVWVWNMD30I9/QA8fKyx</latexit> <latexit sha1_base64="KZAh7y0V0qe3/C6DUlQ3EjB9tIY=">AAAC3XichVFLLwRBEK4Z7/VaXCQuHZsVLpsakRAiES6OXouwTGZaLxPzykzvBps9uoi4OjiRiIif4eIPOLi4iyOJi4Pa2Qlhg5pM99df1Vf9dbfp21YoER8Uta6+obGpuSXR2tbe0Zns6l4OvULARZZ7thesmkYobMsVWWlJW6z6gTAc0xYr5u5MJb9SFEFoee6S3PfFhmNsu1be4oYkSk/u+brDBtkem2A5uSOkwYbYJMvlA4OzEtPKNKxR/jNXZj7bJBLLtbo0+y7kf1XryRRmMApWC7QYpCCOOS95BTnYAg84FMABAS5IwjYYENK3Dhog+MRtQIm4gJAV5QWUIUHaAlUJqjCI3aVxm1brMevSutIzjNScdrHpD0jJII33eI0veIc3+ITvv/YqRT0qXvZpNqta4eudR72Lb/+qHJol7Hyp/vQsIQ9jkVeLvPsRUzkFr+qLB6cvi+ML6dIAXuAz+T/HB7ylE7jFV345LxbOIEEPoP287lqwPJzRMKPNj6SmpuOnaIY+6IdBuu9RmIJZmIMs7fuo1CvtSoeqq4fqsXpSLVWVWNMD30I9/QA8fKyx</latexit> pm(x; ✓) = 1 c p0 m (x; ✓) <latexit sha1_base64="ASnzXrfWOxpQ/pmoxwguySivJ8Y=">AAACpXichVFNSxtBGH6yrV9p1bS9CF4Gg61ewrulYFEKQS+9CH4lCkaX3XGii/vF7iQ0DfkD/oEeerKgpfgzvPTQqwV/gvSo4KWHvtksSBuq7zAzzzzzPu88M+NEnptoosuc8ejxwODQ8Ej+ydPRsfHCs+fVJGzEUlVk6IXxlmMnynMDVdGu9tRWFCvbdzy16Rwudfc3mypO3DDY0K1I7fj2fuDWXWlrpqxCObJ8MSM+iAVR0wdK22JWvBOiVo9tKdrC7PAgRUdEYpcRdfqyrUKRSpSG6AdmBorIYiUsfEUNewgh0YAPhQCasQcbCbdtmCBEzO2gzVzMyE33FTrIs7bBWYozbGYPedzn1XbGBrzu1kxSteRTPO4xKwWm6YK+0TV9pzO6ot//rdVOa3S9tHh2eloVWeNHE+u3D6p8njUO7lT3etao423q1WXvUcp0byF7+ubHT9fr82vT7Zf0hX6x/2O6pHO+QdC8kSerau0z8vwB5r/P3Q+qr0smlczVN8XyYvYVw5jEFGb4vedQxnusoMLnnuIHLvDTeGUsGxtGtZdq5DLNC/wVhvUHufydiw==</latexit> <latexit sha1_base64="ASnzXrfWOxpQ/pmoxwguySivJ8Y=">AAACpXichVFNSxtBGH6yrV9p1bS9CF4Gg61ewrulYFEKQS+9CH4lCkaX3XGii/vF7iQ0DfkD/oEeerKgpfgzvPTQqwV/gvSo4KWHvtksSBuq7zAzzzzzPu88M+NEnptoosuc8ejxwODQ8Ej+ydPRsfHCs+fVJGzEUlVk6IXxlmMnynMDVdGu9tRWFCvbdzy16Rwudfc3mypO3DDY0K1I7fj2fuDWXWlrpqxCObJ8MSM+iAVR0wdK22JWvBOiVo9tKdrC7PAgRUdEYpcRdfqyrUKRSpSG6AdmBorIYiUsfEUNewgh0YAPhQCasQcbCbdtmCBEzO2gzVzMyE33FTrIs7bBWYozbGYPedzn1XbGBrzu1kxSteRTPO4xKwWm6YK+0TV9pzO6ot//rdVOa3S9tHh2eloVWeNHE+u3D6p8njUO7lT3etao423q1WXvUcp0byF7+ubHT9fr82vT7Zf0hX6x/2O6pHO+QdC8kSerau0z8vwB5r/P3Q+qr0smlczVN8XyYvYVw5jEFGb4vedQxnusoMLnnuIHLvDTeGUsGxtGtZdq5DLNC/wVhvUHufydiw==</latexit> <latexit sha1_base64="ASnzXrfWOxpQ/pmoxwguySivJ8Y=">AAACpXichVFNSxtBGH6yrV9p1bS9CF4Gg61ewrulYFEKQS+9CH4lCkaX3XGii/vF7iQ0DfkD/oEeerKgpfgzvPTQqwV/gvSo4KWHvtksSBuq7zAzzzzzPu88M+NEnptoosuc8ejxwODQ8Ej+ydPRsfHCs+fVJGzEUlVk6IXxlmMnynMDVdGu9tRWFCvbdzy16Rwudfc3mypO3DDY0K1I7fj2fuDWXWlrpqxCObJ8MSM+iAVR0wdK22JWvBOiVo9tKdrC7PAgRUdEYpcRdfqyrUKRSpSG6AdmBorIYiUsfEUNewgh0YAPhQCasQcbCbdtmCBEzO2gzVzMyE33FTrIs7bBWYozbGYPedzn1XbGBrzu1kxSteRTPO4xKwWm6YK+0TV9pzO6ot//rdVOa3S9tHh2eloVWeNHE+u3D6p8njUO7lT3etao423q1WXvUcp0byF7+ubHT9fr82vT7Zf0hX6x/2O6pHO+QdC8kSerau0z8vwB5r/P3Q+qr0smlczVN8XyYvYVw5jEFGb4vedQxnusoMLnnuIHLvDTeGUsGxtGtZdq5DLNC/wVhvUHufydiw==</latexit> <latexit sha1_base64="ASnzXrfWOxpQ/pmoxwguySivJ8Y=">AAACpXichVFNSxtBGH6yrV9p1bS9CF4Gg61ewrulYFEKQS+9CH4lCkaX3XGii/vF7iQ0DfkD/oEeerKgpfgzvPTQqwV/gvSo4KWHvtksSBuq7zAzzzzzPu88M+NEnptoosuc8ejxwODQ8Ej+ydPRsfHCs+fVJGzEUlVk6IXxlmMnynMDVdGu9tRWFCvbdzy16Rwudfc3mypO3DDY0K1I7fj2fuDWXWlrpqxCObJ8MSM+iAVR0wdK22JWvBOiVo9tKdrC7PAgRUdEYpcRdfqyrUKRSpSG6AdmBorIYiUsfEUNewgh0YAPhQCasQcbCbdtmCBEzO2gzVzMyE33FTrIs7bBWYozbGYPedzn1XbGBrzu1kxSteRTPO4xKwWm6YK+0TV9pzO6ot//rdVOa3S9tHh2eloVWeNHE+u3D6p8njUO7lT3etao423q1WXvUcp0byF7+ubHT9fr82vT7Zf0hX6x/2O6pHO+QdC8kSerau0z8vwB5r/P3Q+qr0smlczVN8XyYvYVw5jEFGb4vedQxnusoMLnnuIHLvDTeGUsGxtGtZdq5DLNC/wVhvUHufydiw==</latexit> Original: NCE:
  8. Overview of NCE 1. Replace an expensive partition function with

    
 a learnable parameter in a trained model 2. Sample noise data from a noise distribution 3. Classify between observed data and noise data 8
  9. Noise Distribution To classify between observed data and noise data,

    
 NCE’s noise distribution … • has analytical expression for a log PDF/PMF • can be sampled easily • is similar to observed data distribution in some aspect • Ex. Covariance structure for natural images • Ex. Unigram distribution in NLP 9
  10. Overview of NCE 1. Replace an expensive partition function with

    
 a learnable parameter in a trained model 2. Sample noise data from a noise distribution 3. Classify between observed data and noise data 10
  11. Classify between Observed Data and Noise Data Objective function (Bernoulli

    loss) 11 Td X t=1 ln[h(xt; ✓)] + Tn X t=1 ln[1 h(yt; ✓)] <latexit sha1_base64="G/6GZ3Rc5jIp4mNZtxr4wsWZ/zE=">AAAC1XichZHPSxtBFMe/2fqrUWtaL0IvQ0VRpOGtF0tLQRRKj/6KEZK47K6jWdxf7E5C45Kb9NBLjz14sYVSpH9GL/0HWvA/sPSo4EXBl03A1mj7hpl58533efNmxgpdJ1ZExxntXk9vX//A/ezg0PCDkdzDR+txUItsWbADN4g2LDOWruPLgnKUKzfCSJqe5cqitbvY2i/WZRQ7gb+mGqGseOaO72w7tqlYMnJhOa55RqJe6s3NRKwZidgSTW5l1xclURVT4g1ripUXoqyqUpliWlTEjOji/D84/WmKNm5Bjdw45Sk10e3oHWd8/tXJ5SGApSD3BWVsIYCNGjxI+FDsuzARcytBByFkrYKEtYg9J92XaCLLbI2jJEeYrO7yuMOrUkf1ed3KGae0zae43CMmBSboBx3RKX2nr/SLLu7MlaQ5WrU0eLbarAyNkXdjq+f/pTyeFarX1D9rVtjGs7RWh2sPU6V1C7vN1/c+nK4+X5lIJukT/eb6P9IxfeMb+PUz+/OyXDlAlj9Av/nc3c76bF6nvL7MP7GAtg3gMZ5git97DvN4jSUU+NyfuMj0Zvq0otbU9rW37VAt02FG8Zdp768A2SqwkQ==</latexit> <latexit sha1_base64="r/ccuq7XYw1XTNxAkIue4DeX7zQ=">AAAC1XichZHPSxtBFMdf1vor2ibaS8HLoCiW0vDWi6IIYqF4TNQYIYnL7jpJFvcXu5NgusSTePDSowcPoiBF/DO8eBcF/wPFo4IXBV82AbVp9Q0z8+Y77/PmzYzmmoYvEC8jUtuH9o7Oru5oT+/HT7F4X/+S75Q9nad1x3S8ZU31uWnYPC0MYfJl1+OqpZk8o639qO9nKtzzDcdeFFWX5y21aBsFQ1cFSUrczfllSwnEtFxbCdiiErBVVqOWM22WZSU2ytZJE6RMsZwocaGyryzPvrEWzn7Byd9DtPoPVIkPYQJDY62O3HSGZn5ePe6dORtJJ/4HcrAKDuhQBgs42CDIN0EFn1oWZEBwSctDQJpHnhHuc6hBlNgyRXGKUEldo7FIq2xTtWldz+mHtE6nmNQ9IhkM4zke4S2e4jFe48N/cwVhjnotVZq1BstdJbb9ZeH+XcqiWUDpmXqzZgEFmAhrNah2N1Tqt9AbfOXXzu3C5PxwMIIHeEP17+MlntAN7Mqdfpji87sQpQ+Q/37uVmdpLCFjQk7RT8xCw7pgAAZhlN57HGZgDpKQpnMv4CHSHumQMlJN2pS2GqFSpMl8hlcm/X4CB2eyUA==</latexit> <latexit sha1_base64="r/ccuq7XYw1XTNxAkIue4DeX7zQ=">AAAC1XichZHPSxtBFMdf1vor2ibaS8HLoCiW0vDWi6IIYqF4TNQYIYnL7jpJFvcXu5NgusSTePDSowcPoiBF/DO8eBcF/wPFo4IXBV82AbVp9Q0z8+Y77/PmzYzmmoYvEC8jUtuH9o7Oru5oT+/HT7F4X/+S75Q9nad1x3S8ZU31uWnYPC0MYfJl1+OqpZk8o639qO9nKtzzDcdeFFWX5y21aBsFQ1cFSUrczfllSwnEtFxbCdiiErBVVqOWM22WZSU2ytZJE6RMsZwocaGyryzPvrEWzn7Byd9DtPoPVIkPYQJDY62O3HSGZn5ePe6dORtJJ/4HcrAKDuhQBgs42CDIN0EFn1oWZEBwSctDQJpHnhHuc6hBlNgyRXGKUEldo7FIq2xTtWldz+mHtE6nmNQ9IhkM4zke4S2e4jFe48N/cwVhjnotVZq1BstdJbb9ZeH+XcqiWUDpmXqzZgEFmAhrNah2N1Tqt9AbfOXXzu3C5PxwMIIHeEP17+MlntAN7Mqdfpji87sQpQ+Q/37uVmdpLCFjQk7RT8xCw7pgAAZhlN57HGZgDpKQpnMv4CHSHumQMlJN2pS2GqFSpMl8hlcm/X4CB2eyUA==</latexit> <latexit sha1_base64="MUizgpG4hUbY1PLaPd5V/ZORBxQ=">AAAC1XichZE7SwNBEMcn5zu+ojaCzWJQFDHM2SiKINpY+ooRknjcnas5vBd3m2A80omFjaWFlYKI+DFs/AIKfgSxVLCxcHI58BEfs+zu7H/nNzu7q7mm4QvEh5jU0NjU3NLaFm/v6OzqTvT0rvtO0dN5WndMx9vQVJ+bhs3TwhAm33A9rlqayTPa7kJ1P1Pinm849poouzxvqTu2sW3oqiBJSbg5v2gpgZiVK5sBW1MCtsUq1HKmzbKswEbYHmmClBmWEwUuVDbK8myM1XH2J04eD9HyD6iSSGIKQ2P1jhw5SYhsyUlcQg62wAEdimABBxsE+Sao4FPLggwILml5CEjzyDPCfQ4ViBNbpChOESqpuzTu0CobqTatqzn9kNbpFJO6RySDIbzDK3zGW7zGR3z7NVcQ5qjWUqZZq7HcVbqP+ldf/6UsmgUUPqg/axawDVNhrQbV7oZK9RZ6jS/tnzyvTq8MBcN4jk9U/xk+4A3dwC696BfLfOUU4vQB8vfnrnfWJ1IypuRlTM7NR1/RCgMwCCP03pMwB4uwBGk69x7eYk2xZikjVaQD6bAWKsUipg++mHT8DkQkrYY=</latexit> G(x; ✓) = ln pm(x; ✓) ln pn(x) <latexit sha1_base64="nkIHah4WsDUAHBQJ7DokrYE9vWU=">AAACpHichVHLShxBFD22j+j4mNFshGwKRVHE4XY2iiKKLnThQp2MCo4O3W3pNNMvumsGzeAP+AMu3GhARPyMbALZxoWfIFkayMaFd7onBBX1Nl116px7bt2qMgPHjhTRbZPW3NLa9qG9I9XZ1d2TzvT2rUd+JbRk3vIdP9w0jUg6tifzylaO3AxCabimIzfM8kJd36jKMLJ974s6DOS2a+x79p5tGYqpYmZ2UYyIAzEtCqoklSFGxYwoOJ4IijXhiqPn4vg/0Uuk0WJmkLIUh3gJ9AYYnBsojJ0BWPEzlyhgFz4sVOBCwoNi7MBAxN8WdBAC5rZRYy5kZMe6xBFS7K1wluQMg9kyj/u82mqwHq/rNaPYbfEuDv8hOwWG6Iau6J5+0DXd0cOrtWpxjXovhzybiVcGxfRxf+7vuy6XZ4XSf9ebPSvsYTLu1ebeg5ipn8JK/NWvJ/e5qbWh2jB9o9/c/znd0nc+gVf9Y12syrVTpPgB9OfX/RKsf87qlNVX+SXmkUQ7PmEAI3zfE5jDElaQ530v8BO/cKMNa8taTssnqVpTw/MRT0LbeQQxa55+</latexit> <latexit sha1_base64="kzsjsnHuwYrNLdSfmmGNKNs1UBQ=">AAACpHichVHPSxtBFP7cqtXU6tpehF6GiKKI4W0viqU0tAc9eDBJo4HEht111MX9xe4kNA3+Az0LHnqyICL+Gb0UvJpD/oTSo4VeeujLbkqpon3Lznzzfe9782bGCl0nVkTdAe3B4NDww5HRzKOxx+MT+uSTzThoRLYs24EbRBXLjKXr+LKsHOXKShhJ07NcuWUdvOnpW00ZxU7gv1WtUG575p7v7Dq2qZiq669WxZx4L16ImtqXyhTz4qWoub4I623hicOb4uIf0U+l+bo+TTlKQtwGRh9M57O1haNuvrUR6GeoYQcBbDTgQcKHYuzCRMxfFQYIIXPbaDMXMXISXeIQGfY2OEtyhsnsAY97vKr2WZ/XvZpx4rZ5F5f/iJ0CM9Shc7qmr3RB3+jXnbXaSY1eLy2erdQrw/rEx6nSz/+6PJ4V9v+67u1ZYRfLSa8O9x4mTO8Udupvfji+Lq0UZ9qz9Jm+c/8n1KUvfAK/+cM+LcjiJ2T4AYyb130bbD7PGZQzCvwSr5HGCJ4hizm+7yXksYYNlHnfU1ziCh1tVlvXSlo5TdUG+p6n+Ce0d78B972gBA==</latexit> <latexit sha1_base64="kzsjsnHuwYrNLdSfmmGNKNs1UBQ=">AAACpHichVHPSxtBFP7cqtXU6tpehF6GiKKI4W0viqU0tAc9eDBJo4HEht111MX9xe4kNA3+Az0LHnqyICL+Gb0UvJpD/oTSo4VeeujLbkqpon3Lznzzfe9782bGCl0nVkTdAe3B4NDww5HRzKOxx+MT+uSTzThoRLYs24EbRBXLjKXr+LKsHOXKShhJ07NcuWUdvOnpW00ZxU7gv1WtUG575p7v7Dq2qZiq669WxZx4L16ImtqXyhTz4qWoub4I623hicOb4uIf0U+l+bo+TTlKQtwGRh9M57O1haNuvrUR6GeoYQcBbDTgQcKHYuzCRMxfFQYIIXPbaDMXMXISXeIQGfY2OEtyhsnsAY97vKr2WZ/XvZpx4rZ5F5f/iJ0CM9Shc7qmr3RB3+jXnbXaSY1eLy2erdQrw/rEx6nSz/+6PJ4V9v+67u1ZYRfLSa8O9x4mTO8Udupvfji+Lq0UZ9qz9Jm+c/8n1KUvfAK/+cM+LcjiJ2T4AYyb130bbD7PGZQzCvwSr5HGCJ4hizm+7yXksYYNlHnfU1ziCh1tVlvXSlo5TdUG+p6n+Ce0d78B972gBA==</latexit> <latexit sha1_base64="7XKN2W0K+PSt9Bee+eSoyWMMdS8=">AAACpHichVG7ThtBFD0shIdJwECDRDPCAkER624aEAiBoIAiBbYxIGFi7W4GWLEv7Y4twPIP8AMUVCBZKOIzaJDShoJPQJQg0VDker0RApTkrnbmzDn33LkzYwaOHSmi2zatveNDZ1d3T6r346e+/vTA4HrkV0JLFi3f8cNN04ikY3uyqGzlyM0glIZrOnLD3F9q6htVGUa2762pw0Buu8auZ+/YlqGYKqfnl8WEOBCzoqT2pDLEpJgTJccTQbkmXFF/K37+I3otabKczlCW4hDvgZ6ADJJY9dMXKOE7fFiowIWEB8XYgYGIvy3oIATMbaPGXMjIjnWJOlLsrXCW5AyD2X0ed3m1lbAer5s1o9ht8S4O/yE7Bcbohn7QA13TJd3R819r1eIazV4OeTZbXhmU+4+HC0//dbk8K+y9uP7Zs8IOpuNebe49iJnmKayWv3p08lCYyY/Vxumc7rn/M7qlKz6BV320GjmZP0WKH0B/e93vwfqXrE5ZPUeZhcXkKboxglFM8H1PYQErWEWR923gJ37hRhvXvmoFrdhK1doSzxBehfbtN2CBnPU=</latexit> Parametric sigmoid: Log ratio: h(x; ✓) = 1 1 + Tn Td exp( G(x; ✓)) <latexit sha1_base64="Bsnuuk1UBTAi1kaQfN4MvcDfMSw=">AAACp3ichVFNS9xQFD2mfnVqddRNwU1QLDMUhxs3ilIQC9WdnzMOODK8xDdOMJOE5M2ghvwBF2676EYLRaU/oxtX3RX1J5QuLXTTRW8yA/2Q1hvy3rnn3XPfee+ZvmOHiui2S3vU3dPb1/8482Tg6eBQdnikFHrNwJJFy3O8oGyKUDq2K4vKVo4s+4EUDdORW+b+q2R9qyWD0PbcTXXoy52G2HPtmm0JxVQ1u1jPHczrFVWXSuT1l3qlFggrMuLI0F90ks2qG/OwG+sVeeDnpvSl3yT5OFPNTlCB0tDvA6MDJhZez5ycAlj1sueoYBceLDTRgIQLxdiBQMjfNgwQfOZ2EDEXMLLTdYkYGdY2uUpyhWB2n8c9zrY7rMt50jNM1Rbv4vAfsFLHJH2mS7qjK/pAX+jHP3tFaY/EyyHPZlsr/erQ8bON7w+qGjwr1H+p/utZoYbZ1KvN3v2USU5htfWtozd3G3Prk9Fzekdf2f8Z3dJHPoHb+ma9X5Prb5E8gPH3dd8HpemCQQVjjV9iEe3oxxjGkeP7nsEClrGKIu97gU+4xo2W11a0klZul2pdHc0o/ghN/ASCu6LB</latexit> <latexit sha1_base64="5wVZ+BihKzrd504DFBBvARX3mpg=">AAACp3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQs4ZWhUWCvElGSkliRqKtgqxKQVJSZXG9ZWGypoQzkh8Xm1QCKlViEmtaJAQ1fBHUmLZi1XvICygZ4BGChgMgyhDGUHN/O++VmTPwfkCyxniGFIYchnSGYoZchlSGXIYygBsnMYEhmKgTCawZDBgKEAKBbLUA0UKwKyMsHyqQy1DFxAvaVAValAFYlA0WwgmQ7kRUNF84B8kJnFYN3JQFtygLgIqFOBQdXgqsFKg88GJwxWG7w0+IPTrGqwGSC3VALpJIje1IJ4/i6J4O8EdeUC6RKGDIQuvG4uYUhjsAC7NRPo9gKwCMgXyRD9ZVXTPwdbBalWqxksMngNdP9Cg5sGh4E+yCv7krw0MDVoNgMoAgzRgxuTEWakZ2igZxgIjAknBgjgYJBmUGLQAIa3OYMDgwdDAEMo0N4VDOcYrjPcYNJk8mcKY4qAKGVihOoRZkABTIkAB3qksA==</latexit> <latexit sha1_base64="5wVZ+BihKzrd504DFBBvARX3mpg=">AAACp3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQs4ZWhUWCvElGSkliRqKtgqxKQVJSZXG9ZWGypoQzkh8Xm1QCKlViEmtaJAQ1fBHUmLZi1XvICygZ4BGChgMgyhDGUHN/O++VmTPwfkCyxniGFIYchnSGYoZchlSGXIYygBsnMYEhmKgTCawZDBgKEAKBbLUA0UKwKyMsHyqQy1DFxAvaVAValAFYlA0WwgmQ7kRUNF84B8kJnFYN3JQFtygLgIqFOBQdXgqsFKg88GJwxWG7w0+IPTrGqwGSC3VALpJIje1IJ4/i6J4O8EdeUC6RKGDIQuvG4uYUhjsAC7NRPo9gKwCMgXyRD9ZVXTPwdbBalWqxksMngNdP9Cg5sGh4E+yCv7krw0MDVoNgMoAgzRgxuTEWakZ2igZxgIjAknBgjgYJBmUGLQAIa3OYMDgwdDAEMo0N4VDOcYrjPcYNJk8mcKY4qAKGVihOoRZkABTIkAB3qksA==</latexit> <latexit sha1_base64="c/o7XhVG7GfKJA0/DFvy8wegKvo=">AAACp3ichVFNT9tAEH0YWiAtJcAFqReLCJSoajTmAmpVCcGhvZWvhEgERWuzSSwc27I3EdTyH+APcOBEpaqt+jO49NRbVfgJVY9U4sKBsWOpBUQ7lnffvJ03+3bX9B07VETnA9rg0IOHwyOjuUePx56M5ycmq6HXDSxZsTzHC2qmCKVju7KibOXImh9I0TEduWXurSTrWz0ZhLbnbqoDX+50RMu1m7YlFFON/HK7uP9Sr6u2VKKkv9LrzUBYkRFHhv4sSzYbbszDbqzX5b5ffK6//ktSinONfIHKlIZ+FxgZKCCLVS//EXXswoOFLjqQcKEYOxAI+duGAYLP3A4i5gJGdrouESPH2i5XSa4QzO7x2OJsO2NdzpOeYaq2eBeH/4CVOmbpO32mC/pKX+gnXd3bK0p7JF4OeDb7Wuk3xg+nNy7/q+rwrND+o/qnZ4UmFlOvNnv3UyY5hdXX994dXWy8WJ+N5ug9/WL/J3ROp3wCt/fb+rAm14+RPIBx+7rvgup82aCysUaFpeXsKUbwFDMo8n0vYAlvsIoK7/sJ3/ADZ1pJe6tVtVq/VBvINFO4EZq4Bv4VoNY=</latexit> Model’s PDF/PMF Noise PDF/PMF Observed data Noise data
  12. NCE’s Properties NCE has similar properties to MLE • Nonparametric

    estimation • Consistency • Asymptotic normality Check the original paper if you want to know details. 12
  13. Simulations • True data generated by 5-D Gaussian with 0

    mean • Larger number of data is better 13 From Michael U. Gutmann, Aapo Hyvärinen, Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics. JMLR, 2012.
  14. NCE & Neural Probabilistic Language Models [Mnih & Teh, ICML2012]

    • Softmax with large units is a bottleneck in the training time • is set to 1 to avoid storing parameters depend on context words ( ) • Empirically, it did not affect the performance 14 O(|V |#contexts) <latexit sha1_base64="GV40hula22RSeszxaEw3x1HTW14=">AAACgXichVHLLgRBFD3ae7wGGwkLMSFsJrdFQtgIGzvPGRLDpLsVOvqV7poJ2mws/YCFFYkgVvyCjR+w8AliSWJj4U5PJ4LgVqrq1Kl7bp2q0j3LDCTRY5VSXVNbV9/QmGhqbmltS7Z3ZAO34BsiY7iW66/oWiAs0xEZaUpLrHi+0GzdEsv6znR5f7ko/MB0nSW554k1W9tyzE3T0CRT+WTP7OBB9mA9zKVytia3fTs0XEeKXRmUSkP5ZIrSFEXvT6DGIIU45tzkBXLYgAsDBdgQcCAZW9AQcFuFCoLH3BpC5nxGZrQvUEKCtQXOEpyhMbvD4xavVmPW4XW5ZhCpDT7F4u6zshf99EBX9EL3dE1P9P5rrTCqUfayx7Ne0Qov33bUtfj2r8rmWWL7U/WnZ4lNjEVeTfbuRUz5FkZFX9w/flkcX+gPB+iMntn/KT3SHd/AKb4a5/Ni4QQJ/gD1+3P/BNnhtEppdX4kNTkVf0UDutGHQX7vUUxiBnPI8LmHuMQNbpVqZUghZbiSqlTFmk58CWXiAx0DlNg=</latexit> <latexit sha1_base64="GV40hula22RSeszxaEw3x1HTW14=">AAACgXichVHLLgRBFD3ae7wGGwkLMSFsJrdFQtgIGzvPGRLDpLsVOvqV7poJ2mws/YCFFYkgVvyCjR+w8AliSWJj4U5PJ4LgVqrq1Kl7bp2q0j3LDCTRY5VSXVNbV9/QmGhqbmltS7Z3ZAO34BsiY7iW66/oWiAs0xEZaUpLrHi+0GzdEsv6znR5f7ko/MB0nSW554k1W9tyzE3T0CRT+WTP7OBB9mA9zKVytia3fTs0XEeKXRmUSkP5ZIrSFEXvT6DGIIU45tzkBXLYgAsDBdgQcCAZW9AQcFuFCoLH3BpC5nxGZrQvUEKCtQXOEpyhMbvD4xavVmPW4XW5ZhCpDT7F4u6zshf99EBX9EL3dE1P9P5rrTCqUfayx7Ne0Qov33bUtfj2r8rmWWL7U/WnZ4lNjEVeTfbuRUz5FkZFX9w/flkcX+gPB+iMntn/KT3SHd/AKb4a5/Ni4QQJ/gD1+3P/BNnhtEppdX4kNTkVf0UDutGHQX7vUUxiBnPI8LmHuMQNbpVqZUghZbiSqlTFmk58CWXiAx0DlNg=</latexit> <latexit sha1_base64="GV40hula22RSeszxaEw3x1HTW14=">AAACgXichVHLLgRBFD3ae7wGGwkLMSFsJrdFQtgIGzvPGRLDpLsVOvqV7poJ2mws/YCFFYkgVvyCjR+w8AliSWJj4U5PJ4LgVqrq1Kl7bp2q0j3LDCTRY5VSXVNbV9/QmGhqbmltS7Z3ZAO34BsiY7iW66/oWiAs0xEZaUpLrHi+0GzdEsv6znR5f7ko/MB0nSW554k1W9tyzE3T0CRT+WTP7OBB9mA9zKVytia3fTs0XEeKXRmUSkP5ZIrSFEXvT6DGIIU45tzkBXLYgAsDBdgQcCAZW9AQcFuFCoLH3BpC5nxGZrQvUEKCtQXOEpyhMbvD4xavVmPW4XW5ZhCpDT7F4u6zshf99EBX9EL3dE1P9P5rrTCqUfayx7Ne0Qov33bUtfj2r8rmWWL7U/WnZ4lNjEVeTfbuRUz5FkZFX9w/flkcX+gPB+iMntn/KT3SHd/AKb4a5/Ni4QQJ/gD1+3P/BNnhtEppdX4kNTkVf0UDutGHQX7vUUxiBnPI8LmHuMQNbpVqZUghZbiSqlTFmk58CWXiAx0DlNg=</latexit> <latexit sha1_base64="GV40hula22RSeszxaEw3x1HTW14=">AAACgXichVHLLgRBFD3ae7wGGwkLMSFsJrdFQtgIGzvPGRLDpLsVOvqV7poJ2mws/YCFFYkgVvyCjR+w8AliSWJj4U5PJ4LgVqrq1Kl7bp2q0j3LDCTRY5VSXVNbV9/QmGhqbmltS7Z3ZAO34BsiY7iW66/oWiAs0xEZaUpLrHi+0GzdEsv6znR5f7ko/MB0nSW554k1W9tyzE3T0CRT+WTP7OBB9mA9zKVytia3fTs0XEeKXRmUSkP5ZIrSFEXvT6DGIIU45tzkBXLYgAsDBdgQcCAZW9AQcFuFCoLH3BpC5nxGZrQvUEKCtQXOEpyhMbvD4xavVmPW4XW5ZhCpDT7F4u6zshf99EBX9EL3dE1P9P5rrTCqUfayx7Ne0Qov33bUtfj2r8rmWWL7U/WnZ4lNjEVeTfbuRUz5FkZFX9w/flkcX+gPB+iMntn/KT3SHd/AKb4a5/Ni4QQJ/gD1+3P/BNnhtEppdX4kNTkVf0UDutGHQX7vUUxiBnPI8LmHuMQNbpVqZUghZbiSqlTFmk58CWXiAx0DlNg=</latexit> From Andriy Mnih, Yee Whye Teh, A Fast and Simple Algorithm for Training Neural Probabilistic Language Models. ICML, 2012. c <latexit sha1_base64="lu/Cf4S7fI2pX3rx5Qu1Ikpy69Q=">AAACZHichVHLSsNAFD2Nr1ofrYogCFIsFVflRgTFVdGNS9taW1CRJE5raJqEJC3U4g/oVnHhSkFE/Aw3/oALf0AQlxXcuPA2DYgW9Q4zc+bMPXfOzKi2obse0VNI6unt6x8ID0aGhkdGo7Gx8S3XqjmayGuWYTlFVXGFoZsi7+meIYq2I5SqaoiCWllr7xfqwnF1y9z0GrbYrSplUy/pmuIxldH2YglKkR/xbiAHIIEgNqzYDXawDwsaaqhCwITH2IACl9s2ZBBs5nbRZM5hpPv7AkeIsLbGWYIzFGYrPJZ5tR2wJq/bNV1frfEpBneHlXEk6ZFuqUUPdEcv9PFrraZfo+2lwbPa0Qp7L3o8lXv/V1Xl2cPBl+pPzx5KWPa96uzd9pn2LbSOvn543sqtZJPNObqiV/Z/SU90zzcw62/adUZkLxDhD5B/Pnc32FpIyZSSM4uJ9GrwFWFMYxbz/N5LSGMdG8jzuQInOMVZ6FkaliakyU6qFAo0E/gW0swnzEyJ4w==</latexit> <latexit sha1_base64="lu/Cf4S7fI2pX3rx5Qu1Ikpy69Q=">AAACZHichVHLSsNAFD2Nr1ofrYogCFIsFVflRgTFVdGNS9taW1CRJE5raJqEJC3U4g/oVnHhSkFE/Aw3/oALf0AQlxXcuPA2DYgW9Q4zc+bMPXfOzKi2obse0VNI6unt6x8ID0aGhkdGo7Gx8S3XqjmayGuWYTlFVXGFoZsi7+meIYq2I5SqaoiCWllr7xfqwnF1y9z0GrbYrSplUy/pmuIxldH2YglKkR/xbiAHIIEgNqzYDXawDwsaaqhCwITH2IACl9s2ZBBs5nbRZM5hpPv7AkeIsLbGWYIzFGYrPJZ5tR2wJq/bNV1frfEpBneHlXEk6ZFuqUUPdEcv9PFrraZfo+2lwbPa0Qp7L3o8lXv/V1Xl2cPBl+pPzx5KWPa96uzd9pn2LbSOvn543sqtZJPNObqiV/Z/SU90zzcw62/adUZkLxDhD5B/Pnc32FpIyZSSM4uJ9GrwFWFMYxbz/N5LSGMdG8jzuQInOMVZ6FkaliakyU6qFAo0E/gW0swnzEyJ4w==</latexit> <latexit sha1_base64="lu/Cf4S7fI2pX3rx5Qu1Ikpy69Q=">AAACZHichVHLSsNAFD2Nr1ofrYogCFIsFVflRgTFVdGNS9taW1CRJE5raJqEJC3U4g/oVnHhSkFE/Aw3/oALf0AQlxXcuPA2DYgW9Q4zc+bMPXfOzKi2obse0VNI6unt6x8ID0aGhkdGo7Gx8S3XqjmayGuWYTlFVXGFoZsi7+meIYq2I5SqaoiCWllr7xfqwnF1y9z0GrbYrSplUy/pmuIxldH2YglKkR/xbiAHIIEgNqzYDXawDwsaaqhCwITH2IACl9s2ZBBs5nbRZM5hpPv7AkeIsLbGWYIzFGYrPJZ5tR2wJq/bNV1frfEpBneHlXEk6ZFuqUUPdEcv9PFrraZfo+2lwbPa0Qp7L3o8lXv/V1Xl2cPBl+pPzx5KWPa96uzd9pn2LbSOvn543sqtZJPNObqiV/Z/SU90zzcw62/adUZkLxDhD5B/Pnc32FpIyZSSM4uJ9GrwFWFMYxbz/N5LSGMdG8jzuQInOMVZ6FkaliakyU6qFAo0E/gW0swnzEyJ4w==</latexit> <latexit sha1_base64="lu/Cf4S7fI2pX3rx5Qu1Ikpy69Q=">AAACZHichVHLSsNAFD2Nr1ofrYogCFIsFVflRgTFVdGNS9taW1CRJE5raJqEJC3U4g/oVnHhSkFE/Aw3/oALf0AQlxXcuPA2DYgW9Q4zc+bMPXfOzKi2obse0VNI6unt6x8ID0aGhkdGo7Gx8S3XqjmayGuWYTlFVXGFoZsi7+meIYq2I5SqaoiCWllr7xfqwnF1y9z0GrbYrSplUy/pmuIxldH2YglKkR/xbiAHIIEgNqzYDXawDwsaaqhCwITH2IACl9s2ZBBs5nbRZM5hpPv7AkeIsLbGWYIzFGYrPJZ5tR2wJq/bNV1frfEpBneHlXEk6ZFuqUUPdEcv9PFrraZfo+2lwbPa0Qp7L3o8lXv/V1Xl2cPBl+pPzx5KWPa96uzd9pn2LbSOvn543sqtZJPNObqiV/Z/SU90zzcw62/adUZkLxDhD5B/Pnc32FpIyZSSM4uJ9GrwFWFMYxbz/N5LSGMdG8jzuQInOMVZ6FkaliakyU6qFAo0E/gW0swnzEyJ4w==</latexit>
  15. Summary • Proposed new estimation method for a model with

    ineffective partition function • Replace partition function then estimate parameters by classifying between observed data and noise data • Useful for many applications 15
  16. Further Reading • Bregman divergence perspective • An extension paper

    • Review paper by authors • Recent paper: Ciwan Ceylan, Michael Gutmann, Conditional Noise-Contrastive Estimation of Unnormalised Models. ICML, 2018. 16
  17. Implementations • Author’s Matlab code • Auto Differential Frameworks •

    Official TensorFlow function • PyTorch issue 17