Student-t Variational Autoencoder for Robust Density Estimation

- IJCAI2018
- Paper is available at https://www.ijcai.org/Proceedings/2018/374
- Code is available at https://github.com/takahashihiroshi/t_vae

Transcript

1. Copyright©2018 NTT corp. All Rights Reserved. Student-t Variational Autoencoder for Robust Density Estimation Hiroshi Takahashi1, Tomoharu Iwata2, Yuki Yamanaka3, Masanori Yamada3, Satoshi Yagi1 1NTT Software Innovation Center 2NTT Communication Science Laboratories 3NTT Secure Platform Laboratories
2. 2 Copyright©2018 NTT corp. All Rights Reserved. If you use the VAE for continuous data, we recommend using the Student-t distribution as the decoder! In short
3. 3 Copyright©2018 NTT corp. All Rights Reserved. • Estimating data distributions is important for AI • especially for image, audio, video, and detection tasks • The VAE is widely used since it can learn the high- dimensional complicated distributions in these tasks • We focus on estimating distributions of continuous data with the VAE [Introduction] Multivariate density estimation Distribution Data
4. 4 Copyright©2018 NTT corp. All Rights Reserved. p✓ (x) = Z p✓ (x | z) p (z) dz <latexit sha1_base64="IpLMlUDG+yYea5KywlapUvSSTlo=">AAAFdHicnVQ9b9NAGH5dGijhoyksSDBEjVIFqKILAoEqIVWUgaWibUgbqS6R7VwSE3/JPoeklicWxA+gAxNIDIiFgYkVCfEHQOqGxAQdi+iSgffOTkvSklZcFN97z73P4+e9D6uOoXuMkE1p5Nho4viJsZPJU6fPnB1PTZxb9mzf1WhJsw3bLauKRw3doiWmM4OWHZcqpmrQFbU5x+dXWtT1dNt6wDoOXTOVuqXXdE1hCFVSz51KIJsKa6i1QGYNypQwlA1aY7ke2g5lV6832OWp27JusfRRCKZe7Y3We3SnP2sXF4BrBtVwbypZSWVInoiW3h8U4iADcVuwJ6TvIEMVbNDABxMoWMAwNkABD3+rUAACDmJrECDmYqSLeQohJJHrYxbFDAXRJj7rOFqNUQvHXNMTbA3fYuDfRWYasuQLeUO2yWfylvwg3X9qBUKDe+lgr0Zc6lTGn10o7hzKMrFn0NhjDfXMoAa3hFcdvTsC4VVoEb+1vrFdnFnKBlPkFdlC/y/JJvmIFVitX9rrRbr0YogfFSu3ccwQUTEjWoNPpEu+ki0/Ryz2rvxkaD0GRnz1dFGRGStwJ+/JT/KBfDuyiiPm/HjfbezpIWo8n4lcX5wQij4CmEe9tngy4asNc2KPeXUB3MCTE52Ru5gfnSkXo/l4T+6jD67DR7yuK8iRRU4dq+M7EMZnro749C72v4rc66BihHFFHjeRzbmGqDaqcnBVHhV3Nh4+bXRmBMsV54PCY7E2plCxcD35e7kjeuC+BQesdr/u327sWGmoF7z5hcF7vj9YvpYvkHxh8Xpm9k78DRiDizAJObznN2EW7sEClLCWrjQpXZWmR38nLiUyiWyUOiLFnPPQ1xL5P0FUYbY=</latexit> <latexit sha1_base64="IpLMlUDG+yYea5KywlapUvSSTlo=">AAAFdHicnVQ9b9NAGH5dGijhoyksSDBEjVIFqKILAoEqIVWUgaWibUgbqS6R7VwSE3/JPoeklicWxA+gAxNIDIiFgYkVCfEHQOqGxAQdi+iSgffOTkvSklZcFN97z73P4+e9D6uOoXuMkE1p5Nho4viJsZPJU6fPnB1PTZxb9mzf1WhJsw3bLauKRw3doiWmM4OWHZcqpmrQFbU5x+dXWtT1dNt6wDoOXTOVuqXXdE1hCFVSz51KIJsKa6i1QGYNypQwlA1aY7ke2g5lV6832OWp27JusfRRCKZe7Y3We3SnP2sXF4BrBtVwbypZSWVInoiW3h8U4iADcVuwJ6TvIEMVbNDABxMoWMAwNkABD3+rUAACDmJrECDmYqSLeQohJJHrYxbFDAXRJj7rOFqNUQvHXNMTbA3fYuDfRWYasuQLeUO2yWfylvwg3X9qBUKDe+lgr0Zc6lTGn10o7hzKMrFn0NhjDfXMoAa3hFcdvTsC4VVoEb+1vrFdnFnKBlPkFdlC/y/JJvmIFVitX9rrRbr0YogfFSu3ccwQUTEjWoNPpEu+ki0/Ryz2rvxkaD0GRnz1dFGRGStwJ+/JT/KBfDuyiiPm/HjfbezpIWo8n4lcX5wQij4CmEe9tngy4asNc2KPeXUB3MCTE52Ru5gfnSkXo/l4T+6jD67DR7yuK8iRRU4dq+M7EMZnro749C72v4rc66BihHFFHjeRzbmGqDaqcnBVHhV3Nh4+bXRmBMsV54PCY7E2plCxcD35e7kjeuC+BQesdr/u327sWGmoF7z5hcF7vj9YvpYvkHxh8Xpm9k78DRiDizAJObznN2EW7sEClLCWrjQpXZWmR38nLiUyiWyUOiLFnPPQ1xL5P0FUYbY=</latexit> <latexit sha1_base64="IpLMlUDG+yYea5KywlapUvSSTlo=">AAAFdHicnVQ9b9NAGH5dGijhoyksSDBEjVIFqKILAoEqIVWUgaWibUgbqS6R7VwSE3/JPoeklicWxA+gAxNIDIiFgYkVCfEHQOqGxAQdi+iSgffOTkvSklZcFN97z73P4+e9D6uOoXuMkE1p5Nho4viJsZPJU6fPnB1PTZxb9mzf1WhJsw3bLauKRw3doiWmM4OWHZcqpmrQFbU5x+dXWtT1dNt6wDoOXTOVuqXXdE1hCFVSz51KIJsKa6i1QGYNypQwlA1aY7ke2g5lV6832OWp27JusfRRCKZe7Y3We3SnP2sXF4BrBtVwbypZSWVInoiW3h8U4iADcVuwJ6TvIEMVbNDABxMoWMAwNkABD3+rUAACDmJrECDmYqSLeQohJJHrYxbFDAXRJj7rOFqNUQvHXNMTbA3fYuDfRWYasuQLeUO2yWfylvwg3X9qBUKDe+lgr0Zc6lTGn10o7hzKMrFn0NhjDfXMoAa3hFcdvTsC4VVoEb+1vrFdnFnKBlPkFdlC/y/JJvmIFVitX9rrRbr0YogfFSu3ccwQUTEjWoNPpEu+ki0/Ryz2rvxkaD0GRnz1dFGRGStwJ+/JT/KBfDuyiiPm/HjfbezpIWo8n4lcX5wQij4CmEe9tngy4asNc2KPeXUB3MCTE52Ru5gfnSkXo/l4T+6jD67DR7yuK8iRRU4dq+M7EMZnro749C72v4rc66BihHFFHjeRzbmGqDaqcnBVHhV3Nh4+bXRmBMsV54PCY7E2plCxcD35e7kjeuC+BQesdr/u327sWGmoF7z5hcF7vj9YvpYvkHxh8Xpm9k78DRiDizAJObznN2EW7sEClLCWrjQpXZWmR38nLiUyiWyUOiLFnPPQ1xL5P0FUYbY=</latexit> <latexit sha1_base64="IpLMlUDG+yYea5KywlapUvSSTlo=">AAAFdHicnVQ9b9NAGH5dGijhoyksSDBEjVIFqKILAoEqIVWUgaWibUgbqS6R7VwSE3/JPoeklicWxA+gAxNIDIiFgYkVCfEHQOqGxAQdi+iSgffOTkvSklZcFN97z73P4+e9D6uOoXuMkE1p5Nho4viJsZPJU6fPnB1PTZxb9mzf1WhJsw3bLauKRw3doiWmM4OWHZcqpmrQFbU5x+dXWtT1dNt6wDoOXTOVuqXXdE1hCFVSz51KIJsKa6i1QGYNypQwlA1aY7ke2g5lV6832OWp27JusfRRCKZe7Y3We3SnP2sXF4BrBtVwbypZSWVInoiW3h8U4iADcVuwJ6TvIEMVbNDABxMoWMAwNkABD3+rUAACDmJrECDmYqSLeQohJJHrYxbFDAXRJj7rOFqNUQvHXNMTbA3fYuDfRWYasuQLeUO2yWfylvwg3X9qBUKDe+lgr0Zc6lTGn10o7hzKMrFn0NhjDfXMoAa3hFcdvTsC4VVoEb+1vrFdnFnKBlPkFdlC/y/JJvmIFVitX9rrRbr0YogfFSu3ccwQUTEjWoNPpEu+ki0/Ryz2rvxkaD0GRnz1dFGRGStwJ+/JT/KBfDuyiiPm/HjfbezpIWo8n4lcX5wQij4CmEe9tngy4asNc2KPeXUB3MCTE52Ru5gfnSkXo/l4T+6jD67DR7yuK8iRRU4dq+M7EMZnro749C72v4rc66BihHFFHjeRzbmGqDaqcnBVHhV3Nh4+bXRmBMsV54PCY7E2plCxcD35e7kjeuC+BQesdr/u327sWGmoF7z5hcF7vj9YvpYvkHxh8Xpm9k78DRiDizAJObznN2EW7sEClLCWrjQpXZWmR38nLiUyiWyUOiLFnPPQ1xL5P0FUYbY=</latexit> [Preliminary] Variational Autoencoders (VAE) (1/3) • The VAE estimates the probability of a continuous data point x by using latent variable 𝐳: • The log marginal likelihood of VAE is bounded below by the evidence lower bound (ELBO): • The VAE is trained to maximize the sum of ELBO decoder prior encoder ln p✓ (x) E q (z|x) [ln p✓ (x | z)] DKL (q (z | x) kp (z)) <latexit sha1_base64="TMHHLEtnwLscZ1zIOfiQYMCNOig=">AAAGC3icnVTNaxNBFH9bjdb40VYvgpdgaGilhokoSk/FVhC02A+bBro17G4nmzX71d1JTbLsyYsInjz04EnBg3gR9GLxZFH8BxR6EzxpjxV66cE3s2tNk5qWbsjOmzfv95vfe/NmVdc0fEbImtR16HDiyNHuY8njJ06e6untO533naqn0RnNMR2voCo+NQ2bzjCDmbTgelSxVJPOqpVRvj67RD3fcOy7rO7SeUvRbaNkaApDV7FPGsrIpp1yi4FsKayslgKZlSlTwlA2aYkN/PXWQtkz9DIblOVkRtbpYrSgBjfCYrDYhHbLRiu2EcqWsdDGFEXN7Xf7JorGthgxzF8cKwa3bseQg4mR89RjKbcttnmfwWSxN02yRDypdiMXG2mInwmnT/oBMiyAAxpUwQIKNjC0TVDAx98c5ICAi755CNDnoWWIdQohJBFbxSiKEQp6K/jWcTYXe22cc05foDXcxcS/h8gU9JOv5BXZIF/Ia/KTbP2XKxAcXEsdRzXCUrfY8/js9OaeKAtHBuV/qI6aGZTgmtBqoHZXeHgWWoRfaixvTA9P9QcZ8oKso/7nZI2sYgb20m/t5SSdetZBj4qZOzhn6FExIqrBZ7JFvpH16gCx2ZvCw475mGjx6hkiIytm4ErekV9khXzfN4sr1qrxuTs40j3YeDwTsVXRIRR1BDCOfDXxZkJXDUbFGfPsAriCnRP1yBjGRz3loTUen8kd1MF5+IzndQExsojRMTt+AmHcczr6h7Z9B2XkWlsZIx9n5HYF0RxrimyjLFurcn96c/neo3J9WKA80R8UHojaWILFxnryfbkiuuu5BbtUeydvsxonZuqoBW9+rvWetxv5S9kcyeYmL6dHrsffgG44B+dhAO/5VRiBmzABM6BJT6X30kdpNfEk8TaxkvgQhXZJMeYM7HgSn/4Agn6fTg==</latexit> <latexit sha1_base64="TMHHLEtnwLscZ1zIOfiQYMCNOig=">AAAGC3icnVTNaxNBFH9bjdb40VYvgpdgaGilhokoSk/FVhC02A+bBro17G4nmzX71d1JTbLsyYsInjz04EnBg3gR9GLxZFH8BxR6EzxpjxV66cE3s2tNk5qWbsjOmzfv95vfe/NmVdc0fEbImtR16HDiyNHuY8njJ06e6untO533naqn0RnNMR2voCo+NQ2bzjCDmbTgelSxVJPOqpVRvj67RD3fcOy7rO7SeUvRbaNkaApDV7FPGsrIpp1yi4FsKayslgKZlSlTwlA2aYkN/PXWQtkz9DIblOVkRtbpYrSgBjfCYrDYhHbLRiu2EcqWsdDGFEXN7Xf7JorGthgxzF8cKwa3bseQg4mR89RjKbcttnmfwWSxN02yRDypdiMXG2mInwmnT/oBMiyAAxpUwQIKNjC0TVDAx98c5ICAi755CNDnoWWIdQohJBFbxSiKEQp6K/jWcTYXe22cc05foDXcxcS/h8gU9JOv5BXZIF/Ia/KTbP2XKxAcXEsdRzXCUrfY8/js9OaeKAtHBuV/qI6aGZTgmtBqoHZXeHgWWoRfaixvTA9P9QcZ8oKso/7nZI2sYgb20m/t5SSdetZBj4qZOzhn6FExIqrBZ7JFvpH16gCx2ZvCw475mGjx6hkiIytm4ErekV9khXzfN4sr1qrxuTs40j3YeDwTsVXRIRR1BDCOfDXxZkJXDUbFGfPsAriCnRP1yBjGRz3loTUen8kd1MF5+IzndQExsojRMTt+AmHcczr6h7Z9B2XkWlsZIx9n5HYF0RxrimyjLFurcn96c/neo3J9WKA80R8UHojaWILFxnryfbkiuuu5BbtUeydvsxonZuqoBW9+rvWetxv5S9kcyeYmL6dHrsffgG44B+dhAO/5VRiBmzABM6BJT6X30kdpNfEk8TaxkvgQhXZJMeYM7HgSn/4Agn6fTg==</latexit> <latexit sha1_base64="TMHHLEtnwLscZ1zIOfiQYMCNOig=">AAAGC3icnVTNaxNBFH9bjdb40VYvgpdgaGilhokoSk/FVhC02A+bBro17G4nmzX71d1JTbLsyYsInjz04EnBg3gR9GLxZFH8BxR6EzxpjxV66cE3s2tNk5qWbsjOmzfv95vfe/NmVdc0fEbImtR16HDiyNHuY8njJ06e6untO533naqn0RnNMR2voCo+NQ2bzjCDmbTgelSxVJPOqpVRvj67RD3fcOy7rO7SeUvRbaNkaApDV7FPGsrIpp1yi4FsKayslgKZlSlTwlA2aYkN/PXWQtkz9DIblOVkRtbpYrSgBjfCYrDYhHbLRiu2EcqWsdDGFEXN7Xf7JorGthgxzF8cKwa3bseQg4mR89RjKbcttnmfwWSxN02yRDypdiMXG2mInwmnT/oBMiyAAxpUwQIKNjC0TVDAx98c5ICAi755CNDnoWWIdQohJBFbxSiKEQp6K/jWcTYXe22cc05foDXcxcS/h8gU9JOv5BXZIF/Ia/KTbP2XKxAcXEsdRzXCUrfY8/js9OaeKAtHBuV/qI6aGZTgmtBqoHZXeHgWWoRfaixvTA9P9QcZ8oKso/7nZI2sYgb20m/t5SSdetZBj4qZOzhn6FExIqrBZ7JFvpH16gCx2ZvCw475mGjx6hkiIytm4ErekV9khXzfN4sr1qrxuTs40j3YeDwTsVXRIRR1BDCOfDXxZkJXDUbFGfPsAriCnRP1yBjGRz3loTUen8kd1MF5+IzndQExsojRMTt+AmHcczr6h7Z9B2XkWlsZIx9n5HYF0RxrimyjLFurcn96c/neo3J9WKA80R8UHojaWILFxnryfbkiuuu5BbtUeydvsxonZuqoBW9+rvWetxv5S9kcyeYmL6dHrsffgG44B+dhAO/5VRiBmzABM6BJT6X30kdpNfEk8TaxkvgQhXZJMeYM7HgSn/4Agn6fTg==</latexit> <latexit sha1_base64="TMHHLEtnwLscZ1zIOfiQYMCNOig=">AAAGC3icnVTNaxNBFH9bjdb40VYvgpdgaGilhokoSk/FVhC02A+bBro17G4nmzX71d1JTbLsyYsInjz04EnBg3gR9GLxZFH8BxR6EzxpjxV66cE3s2tNk5qWbsjOmzfv95vfe/NmVdc0fEbImtR16HDiyNHuY8njJ06e6untO533naqn0RnNMR2voCo+NQ2bzjCDmbTgelSxVJPOqpVRvj67RD3fcOy7rO7SeUvRbaNkaApDV7FPGsrIpp1yi4FsKayslgKZlSlTwlA2aYkN/PXWQtkz9DIblOVkRtbpYrSgBjfCYrDYhHbLRiu2EcqWsdDGFEXN7Xf7JorGthgxzF8cKwa3bseQg4mR89RjKbcttnmfwWSxN02yRDypdiMXG2mInwmnT/oBMiyAAxpUwQIKNjC0TVDAx98c5ICAi755CNDnoWWIdQohJBFbxSiKEQp6K/jWcTYXe22cc05foDXcxcS/h8gU9JOv5BXZIF/Ia/KTbP2XKxAcXEsdRzXCUrfY8/js9OaeKAtHBuV/qI6aGZTgmtBqoHZXeHgWWoRfaixvTA9P9QcZ8oKso/7nZI2sYgb20m/t5SSdetZBj4qZOzhn6FExIqrBZ7JFvpH16gCx2ZvCw475mGjx6hkiIytm4ErekV9khXzfN4sr1qrxuTs40j3YeDwTsVXRIRR1BDCOfDXxZkJXDUbFGfPsAriCnRP1yBjGRz3loTUen8kd1MF5+IzndQExsojRMTt+AmHcczr6h7Z9B2XkWlsZIx9n5HYF0RxrimyjLFurcn96c/neo3J9WKA80R8UHojaWILFxnryfbkiuuu5BbtUeydvsxonZuqoBW9+rvWetxv5S9kcyeYmL6dHrsffgG44B+dhAO/5VRiBmzABM6BJT6X30kdpNfEk8TaxkvgQhXZJMeYM7HgSn/4Agn6fTg==</latexit>
5. 5 Copyright©2018 NTT corp. All Rights Reserved. • For continuous data, the encoder, decoder, and prior distributions are usually modeled by a Gaussian: [Preliminary] Variational Autoencoders (VAE) (2/3) p (z) = N (z|0, I) <latexit sha1_base64="FTYnFVYe5jjPP7/aoCke9jUj678=">AAAFL3icnVTPaxNBFH6pq631R1O9CF6KoRKlhJeiKAWhWA96qDaNaQNNDbvrJFmzv9idjU3XnLyI/0APnhQ8iBcFQfAqiP+ASg9KQVBqjxV66aFvZjdq05qKE7Lz5s37vv3emzeruabhc8TlRM8+Zf+B3r6D/YcOHzk6kBw8NuM7gaezgu6YjlfUVJ+Zhs0K3OAmK7oeUy3NZLNafULszzaY5xuOfZM3XTZvqVXbqBi6yslVTubckskqPF2yVF7TKuFiq+QZ1Ro/c0l6dNUMr7c6Q+61TWyNtM1rbWA5mcIMyjG008jGRgriMeUMJr5CCW6DAzoEYAEDGzjZJqjg028OsoDgkm8eQvJ5ZBlyn0EL+gkbUBSjCJW8dXpWaTUXe21aC05fonV6i0l/j5BDMIwf8Bmu43t8jqu4+VeuUHIILU2atQjL3PLAwxP5jT1RFs0car9RXTVzqMBFqdUg7a70iCz0CN9YXFrPj00Ph6fxCa6R/se4jG8pA7vxU3+aY9OPuujRKHOH1pw8GkVENXiHm/gR14I02vxF8X7XfEyyRPUMmZEVMwglr/AHvsGVf2Zx5V4Qn7tDM9uDTcRzGRvIDmGkI4RJ4luQTy51LcCEPGORXQjnqXOiHrlC8VFPeWRNxmdyg3QIHrESeZ0lTEnGVCk7cQKtuOeq5B/55ftfRqG1kzHyCUZh1wktsKbMNsqysyp38htLtx7UmmMS5cn+YHBX1saSLDbVU7xXKGK7nlu4S7W38/6pxomZumqhm5/tvOc7jZnRTBYz2dy51Pjl+BvQByfhFKTpnl+AcbgKU1CgXF7DZ/gG35WXyifli7IShfYkYsxx2DaU1S2IiUnM</latexit> <latexit sha1_base64="FTYnFVYe5jjPP7/aoCke9jUj678=">AAAFL3icnVTPaxNBFH6pq631R1O9CF6KoRKlhJeiKAWhWA96qDaNaQNNDbvrJFmzv9idjU3XnLyI/0APnhQ8iBcFQfAqiP+ASg9KQVBqjxV66aFvZjdq05qKE7Lz5s37vv3emzeruabhc8TlRM8+Zf+B3r6D/YcOHzk6kBw8NuM7gaezgu6YjlfUVJ+Zhs0K3OAmK7oeUy3NZLNafULszzaY5xuOfZM3XTZvqVXbqBi6yslVTubckskqPF2yVF7TKuFiq+QZ1Ro/c0l6dNUMr7c6Q+61TWyNtM1rbWA5mcIMyjG008jGRgriMeUMJr5CCW6DAzoEYAEDGzjZJqjg028OsoDgkm8eQvJ5ZBlyn0EL+gkbUBSjCJW8dXpWaTUXe21aC05fonV6i0l/j5BDMIwf8Bmu43t8jqu4+VeuUHIILU2atQjL3PLAwxP5jT1RFs0car9RXTVzqMBFqdUg7a70iCz0CN9YXFrPj00Ph6fxCa6R/se4jG8pA7vxU3+aY9OPuujRKHOH1pw8GkVENXiHm/gR14I02vxF8X7XfEyyRPUMmZEVMwglr/AHvsGVf2Zx5V4Qn7tDM9uDTcRzGRvIDmGkI4RJ4luQTy51LcCEPGORXQjnqXOiHrlC8VFPeWRNxmdyg3QIHrESeZ0lTEnGVCk7cQKtuOeq5B/55ftfRqG1kzHyCUZh1wktsKbMNsqysyp38htLtx7UmmMS5cn+YHBX1saSLDbVU7xXKGK7nlu4S7W38/6pxomZumqhm5/tvOc7jZnRTBYz2dy51Pjl+BvQByfhFKTpnl+AcbgKU1CgXF7DZ/gG35WXyifli7IShfYkYsxx2DaU1S2IiUnM</latexit> <latexit sha1_base64="FTYnFVYe5jjPP7/aoCke9jUj678=">AAAFL3icnVTPaxNBFH6pq631R1O9CF6KoRKlhJeiKAWhWA96qDaNaQNNDbvrJFmzv9idjU3XnLyI/0APnhQ8iBcFQfAqiP+ASg9KQVBqjxV66aFvZjdq05qKE7Lz5s37vv3emzeruabhc8TlRM8+Zf+B3r6D/YcOHzk6kBw8NuM7gaezgu6YjlfUVJ+Zhs0K3OAmK7oeUy3NZLNafULszzaY5xuOfZM3XTZvqVXbqBi6yslVTubckskqPF2yVF7TKuFiq+QZ1Ro/c0l6dNUMr7c6Q+61TWyNtM1rbWA5mcIMyjG008jGRgriMeUMJr5CCW6DAzoEYAEDGzjZJqjg028OsoDgkm8eQvJ5ZBlyn0EL+gkbUBSjCJW8dXpWaTUXe21aC05fonV6i0l/j5BDMIwf8Bmu43t8jqu4+VeuUHIILU2atQjL3PLAwxP5jT1RFs0car9RXTVzqMBFqdUg7a70iCz0CN9YXFrPj00Ph6fxCa6R/se4jG8pA7vxU3+aY9OPuujRKHOH1pw8GkVENXiHm/gR14I02vxF8X7XfEyyRPUMmZEVMwglr/AHvsGVf2Zx5V4Qn7tDM9uDTcRzGRvIDmGkI4RJ4luQTy51LcCEPGORXQjnqXOiHrlC8VFPeWRNxmdyg3QIHrESeZ0lTEnGVCk7cQKtuOeq5B/55ftfRqG1kzHyCUZh1wktsKbMNsqysyp38htLtx7UmmMS5cn+YHBX1saSLDbVU7xXKGK7nlu4S7W38/6pxomZumqhm5/tvOc7jZnRTBYz2dy51Pjl+BvQByfhFKTpnl+AcbgKU1CgXF7DZ/gG35WXyifli7IShfYkYsxx2DaU1S2IiUnM</latexit> <latexit sha1_base64="FTYnFVYe5jjPP7/aoCke9jUj678=">AAAFL3icnVTPaxNBFH6pq631R1O9CF6KoRKlhJeiKAWhWA96qDaNaQNNDbvrJFmzv9idjU3XnLyI/0APnhQ8iBcFQfAqiP+ASg9KQVBqjxV66aFvZjdq05qKE7Lz5s37vv3emzeruabhc8TlRM8+Zf+B3r6D/YcOHzk6kBw8NuM7gaezgu6YjlfUVJ+Zhs0K3OAmK7oeUy3NZLNafULszzaY5xuOfZM3XTZvqVXbqBi6yslVTubckskqPF2yVF7TKuFiq+QZ1Ro/c0l6dNUMr7c6Q+61TWyNtM1rbWA5mcIMyjG008jGRgriMeUMJr5CCW6DAzoEYAEDGzjZJqjg028OsoDgkm8eQvJ5ZBlyn0EL+gkbUBSjCJW8dXpWaTUXe21aC05fonV6i0l/j5BDMIwf8Bmu43t8jqu4+VeuUHIILU2atQjL3PLAwxP5jT1RFs0car9RXTVzqMBFqdUg7a70iCz0CN9YXFrPj00Ph6fxCa6R/se4jG8pA7vxU3+aY9OPuujRKHOH1pw8GkVENXiHm/gR14I02vxF8X7XfEyyRPUMmZEVMwglr/AHvsGVf2Zx5V4Qn7tDM9uDTcRzGRvIDmGkI4RJ4luQTy51LcCEPGORXQjnqXOiHrlC8VFPeWRNxmdyg3QIHrESeZ0lTEnGVCk7cQKtuOeq5B/55ftfRqG1kzHyCUZh1wktsKbMNsqysyp38htLtx7UmmMS5cn+YHBX1saSLDbVU7xXKGK7nlu4S7W38/6pxomZumqhm5/tvOc7jZnRTBYz2dy51Pjl+BvQByfhFKTpnl+AcbgKU1CgXF7DZ/gG35WXyifli7IShfYkYsxx2DaU1S2IiUnM</latexit> : standard Gaussian p✓ (x|z) = N x | µ✓ (z) , 2 ✓ (z) <latexit sha1_base64="LKa5PFN4iOISzfg9th3xwK2OZXg=">AAAFiHicnVTNTxNBFH+LriJ+UPRi4oXYYKohzStRURIjAQ9eUD4skLDQ7C7Tdu1+ZXcWKWtPXoz/AAdPmnhQLx48eTUx/gOacDPxpBwx4cLBN7PbKgWKcZruvPnNe7/9vTdv1vBtK+SIG0rXkaPqsePdJ3pOnjp9pjfTd3Y29KLAZEXTs71g3tBDZlsuK3KL22zeD5juGDabM2rjYn9uhQWh5bkPeN1ni45eca2yZeqcoFLmjV+KNUfnVaMca7zKuN5oaDYr81wTXW08bpprDS2wKlV++ZZETN2O7+3x1hxrWXOiUpNu936LYlALrYqjt9yW4qGDXNOplMliHuXo32sUUiML6Zj0+pTvoMEyeGBCBA4wcIGTbYMOIf0WoAAIPmGLEBMWkGXJfQYN6KHYiLwYeeiE1uhZodVCirq0FpyhjDbpLTb9A4rshwH8gq9xCz/jW/yBOwdyxZJDaKnTbCSxzC/1Pjs/s31olEMzh+qfqI6aOZThhtRqkXZfIiILM4lfWVvfmhmZHogv4UvcJP0vcAM/Ugbuyi/z1RSbft5Bj0GZe7TmhBjkkdTgE+7gV9yMcujyd/NPOuZjkyWqZ8mMnJRBKHmPP/EDfvtnFl/uRem5ezSzQ9iEP5e+kewQRjpimCC+VfnkUtcqjMszFtnFcI06J+mRO+Sf9FRA1kR6JvdJh+ARK5HXFYrRpE+FshMn0Eh7rkL4YAv7X0ahtZ0xwQSjsGsULWJtmW2SZXtVHs5sry89rdZHZFQg+4PBI1kbR7K4VE/xXqGI7Xtu8T7V3s37txovZeqohW5+of2e7zVmh/IFzBemrmZHx9JvQDdcgIuQo3s+DKNwFyahCKaSUa4rt5VRtUdFdVi9mbh2KWnMOdg11LHfhcVolA==</latexit> <latexit sha1_base64="LKa5PFN4iOISzfg9th3xwK2OZXg=">AAAFiHicnVTNTxNBFH+LriJ+UPRi4oXYYKohzStRURIjAQ9eUD4skLDQ7C7Tdu1+ZXcWKWtPXoz/AAdPmnhQLx48eTUx/gOacDPxpBwx4cLBN7PbKgWKcZruvPnNe7/9vTdv1vBtK+SIG0rXkaPqsePdJ3pOnjp9pjfTd3Y29KLAZEXTs71g3tBDZlsuK3KL22zeD5juGDabM2rjYn9uhQWh5bkPeN1ni45eca2yZeqcoFLmjV+KNUfnVaMca7zKuN5oaDYr81wTXW08bpprDS2wKlV++ZZETN2O7+3x1hxrWXOiUpNu936LYlALrYqjt9yW4qGDXNOplMliHuXo32sUUiML6Zj0+pTvoMEyeGBCBA4wcIGTbYMOIf0WoAAIPmGLEBMWkGXJfQYN6KHYiLwYeeiE1uhZodVCirq0FpyhjDbpLTb9A4rshwH8gq9xCz/jW/yBOwdyxZJDaKnTbCSxzC/1Pjs/s31olEMzh+qfqI6aOZThhtRqkXZfIiILM4lfWVvfmhmZHogv4UvcJP0vcAM/Ugbuyi/z1RSbft5Bj0GZe7TmhBjkkdTgE+7gV9yMcujyd/NPOuZjkyWqZ8mMnJRBKHmPP/EDfvtnFl/uRem5ezSzQ9iEP5e+kewQRjpimCC+VfnkUtcqjMszFtnFcI06J+mRO+Sf9FRA1kR6JvdJh+ARK5HXFYrRpE+FshMn0Eh7rkL4YAv7X0ahtZ0xwQSjsGsULWJtmW2SZXtVHs5sry89rdZHZFQg+4PBI1kbR7K4VE/xXqGI7Xtu8T7V3s37txovZeqohW5+of2e7zVmh/IFzBemrmZHx9JvQDdcgIuQo3s+DKNwFyahCKaSUa4rt5VRtUdFdVi9mbh2KWnMOdg11LHfhcVolA==</latexit> <latexit sha1_base64="LKa5PFN4iOISzfg9th3xwK2OZXg=">AAAFiHicnVTNTxNBFH+LriJ+UPRi4oXYYKohzStRURIjAQ9eUD4skLDQ7C7Tdu1+ZXcWKWtPXoz/AAdPmnhQLx48eTUx/gOacDPxpBwx4cLBN7PbKgWKcZruvPnNe7/9vTdv1vBtK+SIG0rXkaPqsePdJ3pOnjp9pjfTd3Y29KLAZEXTs71g3tBDZlsuK3KL22zeD5juGDabM2rjYn9uhQWh5bkPeN1ni45eca2yZeqcoFLmjV+KNUfnVaMca7zKuN5oaDYr81wTXW08bpprDS2wKlV++ZZETN2O7+3x1hxrWXOiUpNu936LYlALrYqjt9yW4qGDXNOplMliHuXo32sUUiML6Zj0+pTvoMEyeGBCBA4wcIGTbYMOIf0WoAAIPmGLEBMWkGXJfQYN6KHYiLwYeeiE1uhZodVCirq0FpyhjDbpLTb9A4rshwH8gq9xCz/jW/yBOwdyxZJDaKnTbCSxzC/1Pjs/s31olEMzh+qfqI6aOZThhtRqkXZfIiILM4lfWVvfmhmZHogv4UvcJP0vcAM/Ugbuyi/z1RSbft5Bj0GZe7TmhBjkkdTgE+7gV9yMcujyd/NPOuZjkyWqZ8mMnJRBKHmPP/EDfvtnFl/uRem5ezSzQ9iEP5e+kewQRjpimCC+VfnkUtcqjMszFtnFcI06J+mRO+Sf9FRA1kR6JvdJh+ARK5HXFYrRpE+FshMn0Eh7rkL4YAv7X0ahtZ0xwQSjsGsULWJtmW2SZXtVHs5sry89rdZHZFQg+4PBI1kbR7K4VE/xXqGI7Xtu8T7V3s37txovZeqohW5+of2e7zVmh/IFzBemrmZHx9JvQDdcgIuQo3s+DKNwFyahCKaSUa4rt5VRtUdFdVi9mbh2KWnMOdg11LHfhcVolA==</latexit> <latexit sha1_base64="LKa5PFN4iOISzfg9th3xwK2OZXg=">AAAFiHicnVTNTxNBFH+LriJ+UPRi4oXYYKohzStRURIjAQ9eUD4skLDQ7C7Tdu1+ZXcWKWtPXoz/AAdPmnhQLx48eTUx/gOacDPxpBwx4cLBN7PbKgWKcZruvPnNe7/9vTdv1vBtK+SIG0rXkaPqsePdJ3pOnjp9pjfTd3Y29KLAZEXTs71g3tBDZlsuK3KL22zeD5juGDabM2rjYn9uhQWh5bkPeN1ni45eca2yZeqcoFLmjV+KNUfnVaMca7zKuN5oaDYr81wTXW08bpprDS2wKlV++ZZETN2O7+3x1hxrWXOiUpNu936LYlALrYqjt9yW4qGDXNOplMliHuXo32sUUiML6Zj0+pTvoMEyeGBCBA4wcIGTbYMOIf0WoAAIPmGLEBMWkGXJfQYN6KHYiLwYeeiE1uhZodVCirq0FpyhjDbpLTb9A4rshwH8gq9xCz/jW/yBOwdyxZJDaKnTbCSxzC/1Pjs/s31olEMzh+qfqI6aOZThhtRqkXZfIiILM4lfWVvfmhmZHogv4UvcJP0vcAM/Ugbuyi/z1RSbft5Bj0GZe7TmhBjkkdTgE+7gV9yMcujyd/NPOuZjkyWqZ8mMnJRBKHmPP/EDfvtnFl/uRem5ezSzQ9iEP5e+kewQRjpimCC+VfnkUtcqjMszFtnFcI06J+mRO+Sf9FRA1kR6JvdJh+ARK5HXFYrRpE+FshMn0Eh7rkL4YAv7X0ahtZ0xwQSjsGsULWJtmW2SZXtVHs5sry89rdZHZFQg+4PBI1kbR7K4VE/xXqGI7Xtu8T7V3s37txovZeqohW5+of2e7zVmh/IFzBemrmZHx9JvQDdcgIuQo3s+DKNwFyahCKaSUa4rt5VRtUdFdVi9mbh2KWnMOdg11LHfhcVolA==</latexit> estimated by neural networks with parameter 𝜽 q (z | x) = N z | µ (x) , 2 (x) <latexit sha1_base64="QaWoTlJjL8pbmYJldLROaywdLxE=">AAAFhXicnVTPTxNBFH6LVhBFil5MvBAbTDXYTBF/hEgk4sELCtRCExaa3WXajt1f7s7Wlk1PXoz/AAdPmhhjvHjw5NXE+A9ows3Ek3LEhAsH38wuaAstxmm68+bN+7793ps3q7sm8zkhG0rPkaOJY719x/tPnBw4NZgcOr3gO4Fn0LzhmI5X0DWfmsymec64SQuuRzVLN+miXp0W+4s16vnMsR/whkuXLa1ssxIzNI6uYvL1o2KoWhqv6KVQdSus2VRNWuLpXd9aU7XY6u6q3lQ9Vq7wi5PSY2hmeK8DIChGfK27ewSjqs/KlhYHrYRjnQLjqZhMkQyRY3i/kY2NFMRj1hlSvoMKq+CAAQFYQMEGjrYJGvj4W4IsEHDRtwwh+jy0mNyn0IR+xAYYRTFCQ28Vn2VcLcVeG9eC05doA99i4t9D5DCMkC/kDdkin8lb8oPsdOQKJYfQ0sBZj7DULQ4+O5vbPhRl4cyh8gfVVTOHEtyQWhlqd6VHZGFE+Nra+lZuYn4kvEBekk3U/4JskI+YgV37Zbyao/PPu+jRMXMH1xw9OkZENfhEdshXshmkic3fFZ50zcdES1SPyYysmEEoeU9+kg/k2z+zuHIviM/dwZkewibiuYwNZIdQ1BHCDPLV5ZNLXXWYlmcssgvhKnZO1CN3MD7qKQ+tmfhM7qMOwSNWIq9LiFFlTBmzEyfQjHuujP7RPd//Mgqt7YyRTzAKu4pogTVltlGW7VV5mNteX3laaUxIlCf7g8JjWRtLsthYT/FeoYgeeG7hAdVu5f1bjRMzddWCNz/bfs/3GwtjmSzJZOfGU1O3429AH5yD85DGe34dpuAuzEIeDGVAuaLcVCYTvYnLifHEtSi0R4kxZ6BlJG79BujJZxA=</latexit> <latexit sha1_base64="QaWoTlJjL8pbmYJldLROaywdLxE=">AAAFhXicnVTPTxNBFH6LVhBFil5MvBAbTDXYTBF/hEgk4sELCtRCExaa3WXajt1f7s7Wlk1PXoz/AAdPmhhjvHjw5NXE+A9ows3Ek3LEhAsH38wuaAstxmm68+bN+7793ps3q7sm8zkhG0rPkaOJY719x/tPnBw4NZgcOr3gO4Fn0LzhmI5X0DWfmsymec64SQuuRzVLN+miXp0W+4s16vnMsR/whkuXLa1ssxIzNI6uYvL1o2KoWhqv6KVQdSus2VRNWuLpXd9aU7XY6u6q3lQ9Vq7wi5PSY2hmeK8DIChGfK27ewSjqs/KlhYHrYRjnQLjqZhMkQyRY3i/kY2NFMRj1hlSvoMKq+CAAQFYQMEGjrYJGvj4W4IsEHDRtwwh+jy0mNyn0IR+xAYYRTFCQ28Vn2VcLcVeG9eC05doA99i4t9D5DCMkC/kDdkin8lb8oPsdOQKJYfQ0sBZj7DULQ4+O5vbPhRl4cyh8gfVVTOHEtyQWhlqd6VHZGFE+Nra+lZuYn4kvEBekk3U/4JskI+YgV37Zbyao/PPu+jRMXMH1xw9OkZENfhEdshXshmkic3fFZ50zcdES1SPyYysmEEoeU9+kg/k2z+zuHIviM/dwZkewibiuYwNZIdQ1BHCDPLV5ZNLXXWYlmcssgvhKnZO1CN3MD7qKQ+tmfhM7qMOwSNWIq9LiFFlTBmzEyfQjHuujP7RPd//Mgqt7YyRTzAKu4pogTVltlGW7VV5mNteX3laaUxIlCf7g8JjWRtLsthYT/FeoYgeeG7hAdVu5f1bjRMzddWCNz/bfs/3GwtjmSzJZOfGU1O3429AH5yD85DGe34dpuAuzEIeDGVAuaLcVCYTvYnLifHEtSi0R4kxZ6BlJG79BujJZxA=</latexit> <latexit sha1_base64="QaWoTlJjL8pbmYJldLROaywdLxE=">AAAFhXicnVTPTxNBFH6LVhBFil5MvBAbTDXYTBF/hEgk4sELCtRCExaa3WXajt1f7s7Wlk1PXoz/AAdPmhhjvHjw5NXE+A9ows3Ek3LEhAsH38wuaAstxmm68+bN+7793ps3q7sm8zkhG0rPkaOJY719x/tPnBw4NZgcOr3gO4Fn0LzhmI5X0DWfmsymec64SQuuRzVLN+miXp0W+4s16vnMsR/whkuXLa1ssxIzNI6uYvL1o2KoWhqv6KVQdSus2VRNWuLpXd9aU7XY6u6q3lQ9Vq7wi5PSY2hmeK8DIChGfK27ewSjqs/KlhYHrYRjnQLjqZhMkQyRY3i/kY2NFMRj1hlSvoMKq+CAAQFYQMEGjrYJGvj4W4IsEHDRtwwh+jy0mNyn0IR+xAYYRTFCQ28Vn2VcLcVeG9eC05doA99i4t9D5DCMkC/kDdkin8lb8oPsdOQKJYfQ0sBZj7DULQ4+O5vbPhRl4cyh8gfVVTOHEtyQWhlqd6VHZGFE+Nra+lZuYn4kvEBekk3U/4JskI+YgV37Zbyao/PPu+jRMXMH1xw9OkZENfhEdshXshmkic3fFZ50zcdES1SPyYysmEEoeU9+kg/k2z+zuHIviM/dwZkewibiuYwNZIdQ1BHCDPLV5ZNLXXWYlmcssgvhKnZO1CN3MD7qKQ+tmfhM7qMOwSNWIq9LiFFlTBmzEyfQjHuujP7RPd//Mgqt7YyRTzAKu4pogTVltlGW7VV5mNteX3laaUxIlCf7g8JjWRtLsthYT/FeoYgeeG7hAdVu5f1bjRMzddWCNz/bfs/3GwtjmSzJZOfGU1O3429AH5yD85DGe34dpuAuzEIeDGVAuaLcVCYTvYnLifHEtSi0R4kxZ6BlJG79BujJZxA=</latexit> <latexit sha1_base64="QaWoTlJjL8pbmYJldLROaywdLxE=">AAAFhXicnVTPTxNBFH6LVhBFil5MvBAbTDXYTBF/hEgk4sELCtRCExaa3WXajt1f7s7Wlk1PXoz/AAdPmhhjvHjw5NXE+A9ows3Ek3LEhAsH38wuaAstxmm68+bN+7793ps3q7sm8zkhG0rPkaOJY719x/tPnBw4NZgcOr3gO4Fn0LzhmI5X0DWfmsymec64SQuuRzVLN+miXp0W+4s16vnMsR/whkuXLa1ssxIzNI6uYvL1o2KoWhqv6KVQdSus2VRNWuLpXd9aU7XY6u6q3lQ9Vq7wi5PSY2hmeK8DIChGfK27ewSjqs/KlhYHrYRjnQLjqZhMkQyRY3i/kY2NFMRj1hlSvoMKq+CAAQFYQMEGjrYJGvj4W4IsEHDRtwwh+jy0mNyn0IR+xAYYRTFCQ28Vn2VcLcVeG9eC05doA99i4t9D5DCMkC/kDdkin8lb8oPsdOQKJYfQ0sBZj7DULQ4+O5vbPhRl4cyh8gfVVTOHEtyQWhlqd6VHZGFE+Nra+lZuYn4kvEBekk3U/4JskI+YgV37Zbyao/PPu+jRMXMH1xw9OkZENfhEdshXshmkic3fFZ50zcdES1SPyYysmEEoeU9+kg/k2z+zuHIviM/dwZkewibiuYwNZIdQ1BHCDPLV5ZNLXXWYlmcssgvhKnZO1CN3MD7qKQ+tmfhM7qMOwSNWIq9LiFFlTBmzEyfQjHuujP7RPd//Mgqt7YyRTzAKu4pogTVltlGW7VV5mNteX3laaUxIlCf7g8JjWRtLsthYT/FeoYgeeG7hAdVu5f1bjRMzddWCNz/bfs/3GwtjmSzJZOfGU1O3429AH5yD85DGe34dpuAuzEIeDGVAuaLcVCYTvYnLifHEtSi0R4kxZ6BlJG79BujJZxA=</latexit> estimated by neural networks with parameter 𝝓
6. 6 Copyright©2018 NTT corp. All Rights Reserved. [Preliminary] Variational Autoencoders (VAE) (3/3) 𝐱 𝝁𝝓 𝐱 𝝈𝝓 𝟐 𝐱 𝝈𝜽 𝟐 𝐳 𝝁𝜽 𝐳 𝐳 𝒒𝝓 𝐳 𝐱 = 𝑵 𝐳; 𝝁𝝓 𝐱 , 𝝈𝝓 𝟐 𝐱 𝒑𝜽(𝐱|𝐳) = 𝑵 𝐱; 𝝁𝜽 𝐳 , 𝝈𝜽 𝟐 𝐳 sampling 𝒑 𝐳 KL 𝝓 𝜽 Diagram of VAE for continuous data
7. 7 Copyright©2018 NTT corp. All Rights Reserved. • When we use the Gaussian as the decoder, the training of VAE often becomes unstable • For example, when we train KDD99 SMTP with VAE, Negative of Mean ELBO sharply jumped up during training [Our problem] Instability of training VAE with Gaussian decoder KDD99 SMTP Dataset 0 200 400 600 800 1000 Number of epochs 0 20 40 60 Negative variational lower bound 981 982 983 984 985 0 20 40 60 Negative of Mean ELBO Negative of Mean ELBO 𝑥 ! 𝑥 " 𝑥# 𝐱 = 𝑥#, 𝑥$, 𝑥% &
8. 8 Copyright©2018 NTT corp. All Rights Reserved. • The cause is division by too small variance in ELBO [Our problem] Cause of this instability: too small variance ln p✓ (x | z) = ln N x | µ✓ (z) , 2 ✓ (z) = X d " (xd µ✓,d (z))2 2 2 ✓,d (z) 1 2 ln 2⇡ 2 ✓,d (z) # <latexit sha1_base64="Jsr/DSIXPPQA7g4EjpoHs36D/4A=">AAAGaHicnVRNTxNBGH5bpSJ+QPWAxkuFQMAAmTYmKokJEQ9eUD6skLDQ7G6n7ch+ZXcWgcmevBj+AAdPmngwXjx48mpi/AOacDPxpJwMJlw4+M7sFmmBom7TnXfeeZ5nnnc+1vAsFnBCNlPpEyfbMqfaT3ecOXvufGdX9sKjwA19kxZN13L9OUMPqMUcWuSMW3TO86luGxadNZbG5fjsMvUD5joP+apHF2y96rAKM3WOqVI2ta5ZTs4rCc3Wec2oCI3XKNejSLNohQ/UsyuRZrNyvbcWaT6r1vhg/21kq6ypW+L+EaSwVFdtHN+TGdICVrX1PdiiKBwFrTcaTh2EdkmUY+T8sFbxdVM0O5CA4X0WhsrHKMu5I1FodISsFp6iZO58pDBOQfPYv/DjZqGj1NVLRoh6cgeDfBL0QvJMutnUN9CgDC6YEIINFBzgGFugQ4C/ecgDAQ9zCyAw52PE1DiFCDqQGyKKIkLH7BK+q9ibT7IO9qVmoNgmzmLh30dmDvrIZ/KabJNP5A35TnaP1BJKQ3pZxdaIudQrda5fmtk5lmVjy6H2h9XSM4cK3FReGXr3VEZWYcb85bWN7ZnR6T7RT16SLfT/gmySD1iBs/zLfDVFp5+38GNg5S72OWYMRMRr8JHski9kKxwgDn8797RlPRZGcvWYqshOFKSTd+QHeU++/rWKp8bCZN9dbOkxahLPFTZUJ4SiDwETqLei3lz5WoFxtceyOgEFPDnxGbmL+PhM+RhNJHvyAH1IHdmTdV1DjqYwVaxO7kCUnLkq5of2cv+rKL02K8Y5qSjjJWRLrqWqjatsXpXHMzsbi89qq6OK5avzQeGJWhtbqTi4nnJe6Ygeum/ikNVu1N3vxk2UWnrBm59vvucHg2Jh5NZIfup679id5BPQDlegBwbwmt+AMbgHk1AEM/Uz3Z3Opa+2bWWyme7M5RiaTiWci9DwZHp+A4X2wBI=</latexit> <latexit sha1_base64="Jsr/DSIXPPQA7g4EjpoHs36D/4A=">AAAGaHicnVRNTxNBGH5bpSJ+QPWAxkuFQMAAmTYmKokJEQ9eUD6skLDQ7G6n7ch+ZXcWgcmevBj+AAdPmngwXjx48mpi/AOacDPxpJwMJlw4+M7sFmmBom7TnXfeeZ5nnnc+1vAsFnBCNlPpEyfbMqfaT3ecOXvufGdX9sKjwA19kxZN13L9OUMPqMUcWuSMW3TO86luGxadNZbG5fjsMvUD5joP+apHF2y96rAKM3WOqVI2ta5ZTs4rCc3Wec2oCI3XKNejSLNohQ/UsyuRZrNyvbcWaT6r1vhg/21kq6ypW+L+EaSwVFdtHN+TGdICVrX1PdiiKBwFrTcaTh2EdkmUY+T8sFbxdVM0O5CA4X0WhsrHKMu5I1FodISsFp6iZO58pDBOQfPYv/DjZqGj1NVLRoh6cgeDfBL0QvJMutnUN9CgDC6YEIINFBzgGFugQ4C/ecgDAQ9zCyAw52PE1DiFCDqQGyKKIkLH7BK+q9ibT7IO9qVmoNgmzmLh30dmDvrIZ/KabJNP5A35TnaP1BJKQ3pZxdaIudQrda5fmtk5lmVjy6H2h9XSM4cK3FReGXr3VEZWYcb85bWN7ZnR6T7RT16SLfT/gmySD1iBs/zLfDVFp5+38GNg5S72OWYMRMRr8JHski9kKxwgDn8797RlPRZGcvWYqshOFKSTd+QHeU++/rWKp8bCZN9dbOkxahLPFTZUJ4SiDwETqLei3lz5WoFxtceyOgEFPDnxGbmL+PhM+RhNJHvyAH1IHdmTdV1DjqYwVaxO7kCUnLkq5of2cv+rKL02K8Y5qSjjJWRLrqWqjatsXpXHMzsbi89qq6OK5avzQeGJWhtbqTi4nnJe6Ygeum/ikNVu1N3vxk2UWnrBm59vvucHg2Jh5NZIfup679id5BPQDlegBwbwmt+AMbgHk1AEM/Uz3Z3Opa+2bWWyme7M5RiaTiWci9DwZHp+A4X2wBI=</latexit> <latexit sha1_base64="Jsr/DSIXPPQA7g4EjpoHs36D/4A=">AAAGaHicnVRNTxNBGH5bpSJ+QPWAxkuFQMAAmTYmKokJEQ9eUD6skLDQ7G6n7ch+ZXcWgcmevBj+AAdPmngwXjx48mpi/AOacDPxpJwMJlw4+M7sFmmBom7TnXfeeZ5nnnc+1vAsFnBCNlPpEyfbMqfaT3ecOXvufGdX9sKjwA19kxZN13L9OUMPqMUcWuSMW3TO86luGxadNZbG5fjsMvUD5joP+apHF2y96rAKM3WOqVI2ta5ZTs4rCc3Wec2oCI3XKNejSLNohQ/UsyuRZrNyvbcWaT6r1vhg/21kq6ypW+L+EaSwVFdtHN+TGdICVrX1PdiiKBwFrTcaTh2EdkmUY+T8sFbxdVM0O5CA4X0WhsrHKMu5I1FodISsFp6iZO58pDBOQfPYv/DjZqGj1NVLRoh6cgeDfBL0QvJMutnUN9CgDC6YEIINFBzgGFugQ4C/ecgDAQ9zCyAw52PE1DiFCDqQGyKKIkLH7BK+q9ibT7IO9qVmoNgmzmLh30dmDvrIZ/KabJNP5A35TnaP1BJKQ3pZxdaIudQrda5fmtk5lmVjy6H2h9XSM4cK3FReGXr3VEZWYcb85bWN7ZnR6T7RT16SLfT/gmySD1iBs/zLfDVFp5+38GNg5S72OWYMRMRr8JHski9kKxwgDn8797RlPRZGcvWYqshOFKSTd+QHeU++/rWKp8bCZN9dbOkxahLPFTZUJ4SiDwETqLei3lz5WoFxtceyOgEFPDnxGbmL+PhM+RhNJHvyAH1IHdmTdV1DjqYwVaxO7kCUnLkq5of2cv+rKL02K8Y5qSjjJWRLrqWqjatsXpXHMzsbi89qq6OK5avzQeGJWhtbqTi4nnJe6Ygeum/ikNVu1N3vxk2UWnrBm59vvucHg2Jh5NZIfup679id5BPQDlegBwbwmt+AMbgHk1AEM/Uz3Z3Opa+2bWWyme7M5RiaTiWci9DwZHp+A4X2wBI=</latexit> <latexit sha1_base64="Jsr/DSIXPPQA7g4EjpoHs36D/4A=">AAAGaHicnVRNTxNBGH5bpSJ+QPWAxkuFQMAAmTYmKokJEQ9eUD6skLDQ7G6n7ch+ZXcWgcmevBj+AAdPmngwXjx48mpi/AOacDPxpJwMJlw4+M7sFmmBom7TnXfeeZ5nnnc+1vAsFnBCNlPpEyfbMqfaT3ecOXvufGdX9sKjwA19kxZN13L9OUMPqMUcWuSMW3TO86luGxadNZbG5fjsMvUD5joP+apHF2y96rAKM3WOqVI2ta5ZTs4rCc3Wec2oCI3XKNejSLNohQ/UsyuRZrNyvbcWaT6r1vhg/21kq6ypW+L+EaSwVFdtHN+TGdICVrX1PdiiKBwFrTcaTh2EdkmUY+T8sFbxdVM0O5CA4X0WhsrHKMu5I1FodISsFp6iZO58pDBOQfPYv/DjZqGj1NVLRoh6cgeDfBL0QvJMutnUN9CgDC6YEIINFBzgGFugQ4C/ecgDAQ9zCyAw52PE1DiFCDqQGyKKIkLH7BK+q9ibT7IO9qVmoNgmzmLh30dmDvrIZ/KabJNP5A35TnaP1BJKQ3pZxdaIudQrda5fmtk5lmVjy6H2h9XSM4cK3FReGXr3VEZWYcb85bWN7ZnR6T7RT16SLfT/gmySD1iBs/zLfDVFp5+38GNg5S72OWYMRMRr8JHski9kKxwgDn8797RlPRZGcvWYqshOFKSTd+QHeU++/rWKp8bCZN9dbOkxahLPFTZUJ4SiDwETqLei3lz5WoFxtceyOgEFPDnxGbmL+PhM+RhNJHvyAH1IHdmTdV1DjqYwVaxO7kCUnLkq5of2cv+rKL02K8Y5qSjjJWRLrqWqjatsXpXHMzsbi89qq6OK5avzQeGJWhtbqTi4nnJe6Ygeum/ikNVu1N3vxk2UWnrBm59vvucHg2Jh5NZIfup679id5BPQDlegBwbwmt+AMbgHk1AEM/Uz3Z3Opa+2bWWyme7M5RiaTiWci9DwZHp+A4X2wBI=</latexit> First term of ELBO 𝑑 : dimension index When the decoded varianve 𝜎' $ 𝐳 is almost zero, this term is sensitive to the error between 𝐱 and its decoded mean 𝜇' 𝐳 the variance of these data points is too small along 𝑥! direction
9. 9 Copyright©2018 NTT corp. All Rights Reserved. [Idea] Introducing a prior for the variance • We can avoid this instability problem by preventing the decoded variance 𝜎' $ 𝐳 from being too small • To penalize small variance, we introduce a Gamma distribution as the prior for the decoded variance 𝜎' $ 𝐳 Gam (⌧ | a, b) = ba⌧a 1 exp ( b⌧) (a) <latexit sha1_base64="Z1r2OJrRE+CLzNWzaUID0Qs+YF0=">AAAFSXicnVTNaxNBFH9pU63xo6leBC/B0BKlDbNFUApCsYJeim1j2kC3DbPbSbJmv9idjYnLnryI/0APnhQ8iBcPnrwKInjyoNCb4El7kgoR6cE3s+tH05qKE7Lz5s37/fb33rxZzTUNnxOymRoYTA8dOjx8JHP02PETI9nRk0u+E3g6K+uO6XgVjfrMNGxW5gY3WcX1GLU0ky1rzVmxv9xinm849k3ecdmqReu2UTN0ytFVzTZVi/KGZ4XXqBWpJqvxgsppoFrGeo5OaKpn1Bv83PhlteZRPdTWQhqJfZwnlUhlbTfGTGoSFUdHoYpsFo236E9vpprNkyKRI7fXUBIjD8mYd0ZTH0GFdXBAhwAsYGADR9sECj7+VkABAi76ViFEn4eWIfcZRJBBbIBRDCMoepv4rONqJfHauBacvkTr+BYT/x4iczBG3pEnZJu8Jk/JJ7LzV65QcggtHZy1GMvc6sj906XugSgLZw6N36i+mjnU4JLUaqB2V3pEFnqMb93Z2C5NL46F4+QR2UL9D8kmeYkZ2K2v+uMFtvigjx4NM3dwzdGjYURcg1dkh7wnW0GB2PxZ5W7ffEy0RPUMmZGVMAglz8ln8oJ8+GcWV+4Fybk7OLMD2EQ8l7GB7BCGOkKYQ762fHKpqw2z8oxFdiFMYefEPXIV4+Oe8tCaS87kBuoQPGIl8jqPGFXG1DE7cQJR0nN19E/88v0vo9Dayxj7BKOwm4gWWFNmG2fZW5Vbpe7G2r1GZ1qiPNkfDG7L2liSxcZ6ivcKRWzfcwv3qfZu3j/VOAlTXy1485Xee77XWJoqKqSoLFzIz1xJvgHDcAbOQgHv+UWYgeswD2XM5S10U5BKpd+kv6S/pb/HoQOpBHMKdo2hwR828FAw</latexit> <latexit sha1_base64="Z1r2OJrRE+CLzNWzaUID0Qs+YF0=">AAAFSXicnVTNaxNBFH9pU63xo6leBC/B0BKlDbNFUApCsYJeim1j2kC3DbPbSbJmv9idjYnLnryI/0APnhQ8iBcPnrwKInjyoNCb4El7kgoR6cE3s+tH05qKE7Lz5s37/fb33rxZzTUNnxOymRoYTA8dOjx8JHP02PETI9nRk0u+E3g6K+uO6XgVjfrMNGxW5gY3WcX1GLU0ky1rzVmxv9xinm849k3ecdmqReu2UTN0ytFVzTZVi/KGZ4XXqBWpJqvxgsppoFrGeo5OaKpn1Bv83PhlteZRPdTWQhqJfZwnlUhlbTfGTGoSFUdHoYpsFo236E9vpprNkyKRI7fXUBIjD8mYd0ZTH0GFdXBAhwAsYGADR9sECj7+VkABAi76ViFEn4eWIfcZRJBBbIBRDCMoepv4rONqJfHauBacvkTr+BYT/x4iczBG3pEnZJu8Jk/JJ7LzV65QcggtHZy1GMvc6sj906XugSgLZw6N36i+mjnU4JLUaqB2V3pEFnqMb93Z2C5NL46F4+QR2UL9D8kmeYkZ2K2v+uMFtvigjx4NM3dwzdGjYURcg1dkh7wnW0GB2PxZ5W7ffEy0RPUMmZGVMAglz8ln8oJ8+GcWV+4Fybk7OLMD2EQ8l7GB7BCGOkKYQ762fHKpqw2z8oxFdiFMYefEPXIV4+Oe8tCaS87kBuoQPGIl8jqPGFXG1DE7cQJR0nN19E/88v0vo9Dayxj7BKOwm4gWWFNmG2fZW5Vbpe7G2r1GZ1qiPNkfDG7L2liSxcZ6ivcKRWzfcwv3qfZu3j/VOAlTXy1485Xee77XWJoqKqSoLFzIz1xJvgHDcAbOQgHv+UWYgeswD2XM5S10U5BKpd+kv6S/pb/HoQOpBHMKdo2hwR828FAw</latexit> <latexit sha1_base64="Z1r2OJrRE+CLzNWzaUID0Qs+YF0=">AAAFSXicnVTNaxNBFH9pU63xo6leBC/B0BKlDbNFUApCsYJeim1j2kC3DbPbSbJmv9idjYnLnryI/0APnhQ8iBcPnrwKInjyoNCb4El7kgoR6cE3s+tH05qKE7Lz5s37/fb33rxZzTUNnxOymRoYTA8dOjx8JHP02PETI9nRk0u+E3g6K+uO6XgVjfrMNGxW5gY3WcX1GLU0ky1rzVmxv9xinm849k3ecdmqReu2UTN0ytFVzTZVi/KGZ4XXqBWpJqvxgsppoFrGeo5OaKpn1Bv83PhlteZRPdTWQhqJfZwnlUhlbTfGTGoSFUdHoYpsFo236E9vpprNkyKRI7fXUBIjD8mYd0ZTH0GFdXBAhwAsYGADR9sECj7+VkABAi76ViFEn4eWIfcZRJBBbIBRDCMoepv4rONqJfHauBacvkTr+BYT/x4iczBG3pEnZJu8Jk/JJ7LzV65QcggtHZy1GMvc6sj906XugSgLZw6N36i+mjnU4JLUaqB2V3pEFnqMb93Z2C5NL46F4+QR2UL9D8kmeYkZ2K2v+uMFtvigjx4NM3dwzdGjYURcg1dkh7wnW0GB2PxZ5W7ffEy0RPUMmZGVMAglz8ln8oJ8+GcWV+4Fybk7OLMD2EQ8l7GB7BCGOkKYQ762fHKpqw2z8oxFdiFMYefEPXIV4+Oe8tCaS87kBuoQPGIl8jqPGFXG1DE7cQJR0nN19E/88v0vo9Dayxj7BKOwm4gWWFNmG2fZW5Vbpe7G2r1GZ1qiPNkfDG7L2liSxcZ6ivcKRWzfcwv3qfZu3j/VOAlTXy1485Xee77XWJoqKqSoLFzIz1xJvgHDcAbOQgHv+UWYgeswD2XM5S10U5BKpd+kv6S/pb/HoQOpBHMKdo2hwR828FAw</latexit> <latexit sha1_base64="Z1r2OJrRE+CLzNWzaUID0Qs+YF0=">AAAFSXicnVTNaxNBFH9pU63xo6leBC/B0BKlDbNFUApCsYJeim1j2kC3DbPbSbJmv9idjYnLnryI/0APnhQ8iBcPnrwKInjyoNCb4El7kgoR6cE3s+tH05qKE7Lz5s37/fb33rxZzTUNnxOymRoYTA8dOjx8JHP02PETI9nRk0u+E3g6K+uO6XgVjfrMNGxW5gY3WcX1GLU0ky1rzVmxv9xinm849k3ecdmqReu2UTN0ytFVzTZVi/KGZ4XXqBWpJqvxgsppoFrGeo5OaKpn1Bv83PhlteZRPdTWQhqJfZwnlUhlbTfGTGoSFUdHoYpsFo236E9vpprNkyKRI7fXUBIjD8mYd0ZTH0GFdXBAhwAsYGADR9sECj7+VkABAi76ViFEn4eWIfcZRJBBbIBRDCMoepv4rONqJfHauBacvkTr+BYT/x4iczBG3pEnZJu8Jk/JJ7LzV65QcggtHZy1GMvc6sj906XugSgLZw6N36i+mjnU4JLUaqB2V3pEFnqMb93Z2C5NL46F4+QR2UL9D8kmeYkZ2K2v+uMFtvigjx4NM3dwzdGjYURcg1dkh7wnW0GB2PxZ5W7ffEy0RPUMmZGVMAglz8ln8oJ8+GcWV+4Fybk7OLMD2EQ8l7GB7BCGOkKYQ762fHKpqw2z8oxFdiFMYefEPXIV4+Oe8tCaS87kBuoQPGIl8jqPGFXG1DE7cQJR0nN19E/88v0vo9Dayxj7BKOwm4gWWFNmG2fZW5Vbpe7G2r1GZ1qiPNkfDG7L2liSxcZ6ivcKRWzfcwv3qfZu3j/VOAlTXy1485Xee77XWJoqKqSoLFzIz1xJvgHDcAbOQgHv+UWYgeswD2XM5S10U5BKpd+kv6S/pb/HoQOpBHMKdo2hwR828FAw</latexit> 𝜏 : the inverse of the variance (1/𝜎$)
10. 10 Copyright©2018 NTT corp. All Rights Reserved. • First, we present the MAP estimation for the VAE • To simplify the calculation, we use Gam(𝜏|1, 𝑏) as the prior • Then, the objective function of MAP estimation is: • However, there are two drawbacks in MAP estimation 1. Tuning 𝑏 is difficult 2. The constant 𝑏 lacks flexibility in density estimation • 𝑏 should depend on a data point [Proposed method] As a simple way: MAP estimation E q (z|x)  ln p✓ (x | z) b 2 ✓ (z) DKL (q (z | x) kp (z)) <latexit sha1_base64="E2zCnb3Er9N2W/INJa05sJ1HI9w=">AAAGCXicnVS9axRBFH8bPY3xI4k2gs1hiMRgwlxQlFTBRBA0mA/zAbfJsruZuxtvv9ydi5cMW9mIVlYprBQsxEZBUFIpAfEfUEgnWGnKCGlS+GZ2T5NLcpHscTtv3rzfb37vzZu1AodFnJBVrenQ4cyRo83HWo6fOHmqta399GTkV0KbTti+44fTlhlRh3l0gjPu0OkgpKZrOXTKKg/K9al5GkbM9+7yhYDOuGbRYwVmmxxdRrvWrbsmL1mWuBEb4r4hkmlB6EGJxbHu0ALvqvkWY91lc7VZNdZDVizxi0lUXne8bLCFgJcoN+spqtsoFmsUPXohNG1hxUKPWNE1jRp8VvTtFFHbVo0zPUOGuHU7jTpYBvokDXk22GOjdGgx2jpIL1FPdqeRS40OSJ8Rv137ATrMgQ82VMAFCh5wtB0wIcJfHnJAIEDfDAj0hWgxtU4hhhbEVjCKYoSJ3jK+izjLp14P55IzUmgbd3HwHyIyC53kK3lF1skX8pr8JJt7cgnFIbUs4GglWBoYrY/Pjm/si3Jx5FD6h2qomUMBrimtDLUHyiOzsBP8/OLS+nj/WKe4QF6QNdT/nKySFczAm/9tvxylY88a6LEwcx/nHD0WRiQ1+Ew2yTeyVukiHn8z/bBhPg5asnpMZeSmDFLJO/KLLJPv/80SqLVKeu4+jnQfNhnPVWxFdQhFHQKGka+q3lzpqsKgOmOZnYAr2DlJjwxhfNJTIVrD6ZncQR2SR85kXt2I0VVMEbOTJxCnPVdE/6W/voMySq31jIlPMkq7jGiJdVS2SZb1Vbk3vrE0+6i00K9QoeoPCg9UbVzF4mE95b5SEd313MQu1d7Ou1WNnzI11II3P1d/z3cak329OdKbG73cMXA9/QY0wzk4D114z6/CANyEEZgAW3uqvdc+ap8yTzJvMx8yy0lok5ZizsC2J7PyB3qvn3U=</latexit> <latexit sha1_base64="E2zCnb3Er9N2W/INJa05sJ1HI9w=">AAAGCXicnVS9axRBFH8bPY3xI4k2gs1hiMRgwlxQlFTBRBA0mA/zAbfJsruZuxtvv9ydi5cMW9mIVlYprBQsxEZBUFIpAfEfUEgnWGnKCGlS+GZ2T5NLcpHscTtv3rzfb37vzZu1AodFnJBVrenQ4cyRo83HWo6fOHmqta399GTkV0KbTti+44fTlhlRh3l0gjPu0OkgpKZrOXTKKg/K9al5GkbM9+7yhYDOuGbRYwVmmxxdRrvWrbsmL1mWuBEb4r4hkmlB6EGJxbHu0ALvqvkWY91lc7VZNdZDVizxi0lUXne8bLCFgJcoN+spqtsoFmsUPXohNG1hxUKPWNE1jRp8VvTtFFHbVo0zPUOGuHU7jTpYBvokDXk22GOjdGgx2jpIL1FPdqeRS40OSJ8Rv137ATrMgQ82VMAFCh5wtB0wIcJfHnJAIEDfDAj0hWgxtU4hhhbEVjCKYoSJ3jK+izjLp14P55IzUmgbd3HwHyIyC53kK3lF1skX8pr8JJt7cgnFIbUs4GglWBoYrY/Pjm/si3Jx5FD6h2qomUMBrimtDLUHyiOzsBP8/OLS+nj/WKe4QF6QNdT/nKySFczAm/9tvxylY88a6LEwcx/nHD0WRiQ1+Ew2yTeyVukiHn8z/bBhPg5asnpMZeSmDFLJO/KLLJPv/80SqLVKeu4+jnQfNhnPVWxFdQhFHQKGka+q3lzpqsKgOmOZnYAr2DlJjwxhfNJTIVrD6ZncQR2SR85kXt2I0VVMEbOTJxCnPVdE/6W/voMySq31jIlPMkq7jGiJdVS2SZb1Vbk3vrE0+6i00K9QoeoPCg9UbVzF4mE95b5SEd313MQu1d7Ou1WNnzI11II3P1d/z3cak329OdKbG73cMXA9/QY0wzk4D114z6/CANyEEZgAW3uqvdc+ap8yTzJvMx8yy0lok5ZizsC2J7PyB3qvn3U=</latexit> <latexit sha1_base64="E2zCnb3Er9N2W/INJa05sJ1HI9w=">AAAGCXicnVS9axRBFH8bPY3xI4k2gs1hiMRgwlxQlFTBRBA0mA/zAbfJsruZuxtvv9ydi5cMW9mIVlYprBQsxEZBUFIpAfEfUEgnWGnKCGlS+GZ2T5NLcpHscTtv3rzfb37vzZu1AodFnJBVrenQ4cyRo83HWo6fOHmqta399GTkV0KbTti+44fTlhlRh3l0gjPu0OkgpKZrOXTKKg/K9al5GkbM9+7yhYDOuGbRYwVmmxxdRrvWrbsmL1mWuBEb4r4hkmlB6EGJxbHu0ALvqvkWY91lc7VZNdZDVizxi0lUXne8bLCFgJcoN+spqtsoFmsUPXohNG1hxUKPWNE1jRp8VvTtFFHbVo0zPUOGuHU7jTpYBvokDXk22GOjdGgx2jpIL1FPdqeRS40OSJ8Rv137ATrMgQ82VMAFCh5wtB0wIcJfHnJAIEDfDAj0hWgxtU4hhhbEVjCKYoSJ3jK+izjLp14P55IzUmgbd3HwHyIyC53kK3lF1skX8pr8JJt7cgnFIbUs4GglWBoYrY/Pjm/si3Jx5FD6h2qomUMBrimtDLUHyiOzsBP8/OLS+nj/WKe4QF6QNdT/nKySFczAm/9tvxylY88a6LEwcx/nHD0WRiQ1+Ew2yTeyVukiHn8z/bBhPg5asnpMZeSmDFLJO/KLLJPv/80SqLVKeu4+jnQfNhnPVWxFdQhFHQKGka+q3lzpqsKgOmOZnYAr2DlJjwxhfNJTIVrD6ZncQR2SR85kXt2I0VVMEbOTJxCnPVdE/6W/voMySq31jIlPMkq7jGiJdVS2SZb1Vbk3vrE0+6i00K9QoeoPCg9UbVzF4mE95b5SEd313MQu1d7Ou1WNnzI11II3P1d/z3cak329OdKbG73cMXA9/QY0wzk4D114z6/CANyEEZgAW3uqvdc+ap8yTzJvMx8yy0lok5ZizsC2J7PyB3qvn3U=</latexit> <latexit sha1_base64="E2zCnb3Er9N2W/INJa05sJ1HI9w=">AAAGCXicnVS9axRBFH8bPY3xI4k2gs1hiMRgwlxQlFTBRBA0mA/zAbfJsruZuxtvv9ydi5cMW9mIVlYprBQsxEZBUFIpAfEfUEgnWGnKCGlS+GZ2T5NLcpHscTtv3rzfb37vzZu1AodFnJBVrenQ4cyRo83HWo6fOHmqta399GTkV0KbTti+44fTlhlRh3l0gjPu0OkgpKZrOXTKKg/K9al5GkbM9+7yhYDOuGbRYwVmmxxdRrvWrbsmL1mWuBEb4r4hkmlB6EGJxbHu0ALvqvkWY91lc7VZNdZDVizxi0lUXne8bLCFgJcoN+spqtsoFmsUPXohNG1hxUKPWNE1jRp8VvTtFFHbVo0zPUOGuHU7jTpYBvokDXk22GOjdGgx2jpIL1FPdqeRS40OSJ8Rv137ATrMgQ82VMAFCh5wtB0wIcJfHnJAIEDfDAj0hWgxtU4hhhbEVjCKYoSJ3jK+izjLp14P55IzUmgbd3HwHyIyC53kK3lF1skX8pr8JJt7cgnFIbUs4GglWBoYrY/Pjm/si3Jx5FD6h2qomUMBrimtDLUHyiOzsBP8/OLS+nj/WKe4QF6QNdT/nKySFczAm/9tvxylY88a6LEwcx/nHD0WRiQ1+Ew2yTeyVukiHn8z/bBhPg5asnpMZeSmDFLJO/KLLJPv/80SqLVKeu4+jnQfNhnPVWxFdQhFHQKGka+q3lzpqsKgOmOZnYAr2DlJjwxhfNJTIVrD6ZncQR2SR85kXt2I0VVMEbOTJxCnPVdE/6W/voMySq31jIlPMkq7jGiJdVS2SZb1Vbk3vrE0+6i00K9QoeoPCg9UbVzF4mE95b5SEd313MQu1d7Ou1WNnzI11II3P1d/z3cak329OdKbG73cMXA9/QY0wzk4D114z6/CANyEEZgAW3uqvdc+ap8yTzJvMx8yy0lok5ZizsC2J7PyB3qvn3U=</latexit> small variance is penalized with regularization parameter 𝑏
11. 11 Copyright©2018 NTT corp. All Rights Reserved. • We propose a more flexible approach by introducing a Gamma prior that depends on latent variables: • By analytically integrating out the 𝜏, we can obtain a Student-t decoder: where [Proposed method] Student-t decoder Gam (⌧ | a (z) , b (z)) <latexit sha1_base64="dHvHwql2rJgZX748PAzRfOjbT2Q=">AAAFMnicnVTPTxNBFH7FVbH+oOjFxEtjg6mGNK9Go+FExEQvRKAUmrDY7C7Tdu3+yu5spWx68mL8Bzh40sSD8eKBE1cT4z+gCZ4wnABvmHDh4JvZjUiBYpymO2/evO/b7715s7pnmQFHXEv1nVJOnznbfy59/sLFSwOZwcszgRv6BisbruX6FV0LmGU6rMxNbrGK5zPN1i02qzfHxP5si/mB6TrTvO2xeVurO2bNNDROrmpmWrU13vDt6JFmd1SL1Xhe5Vqo2uZCVkvWIkKvRUsd1TfrDX5zWD9mI5nS1UwOCyhH9rBRTIwcJGPCHUxtgAoL4IIBIdjAwAFOtgUaBPSbgyIgeOSbh4h8Plmm3GfQgTRhQ4piFKGRt0nPOq3mEq9Da8EZSLRBb7Ho7xMyC0P4Fd/jDn7BD7iJe8dyRZJDaGnTrMdY5lUHXl0t7Z6Ismnm0NhH9dTMoQb3pVaTtHvSI7IwYnxraXmnNDI1FN3At7hN+t/gGn6iDJzWL+PdJJt63UOPTpm7tObk0SkirsFn3MNvuB3m0eEfKy965mORJapnyozshEEoWcEtXMX1f2bx5F6YnLtLMzuBTcRzGRvKDmGkI4Jx4luUTy51LcKYPGORXQR3qXPiHnlI8XFP+WSNJ2fyhHQIHrESed0ijCpj6pSdOIFO0nN18g//8f0vo9DazRj7BKOwm4QWWEtmG2fZXZVnpd3lpy8b7RGJ8mV/MHgua2NLFofqKd4rFLEjzy06otoHef9W4yZMPbXQzS923/PDxsztQhELxck7udEHyTegH67BdcjTPb8Ho/AYJqBMuazCOmzClrKifFd+KBtxaF8qwVyBA0P5+Rt7z0ot</latexit> <latexit sha1_base64="dHvHwql2rJgZX748PAzRfOjbT2Q=">AAAFMnicnVTPTxNBFH7FVbH+oOjFxEtjg6mGNK9Go+FExEQvRKAUmrDY7C7Tdu3+yu5spWx68mL8Bzh40sSD8eKBE1cT4z+gCZ4wnABvmHDh4JvZjUiBYpymO2/evO/b7715s7pnmQFHXEv1nVJOnznbfy59/sLFSwOZwcszgRv6BisbruX6FV0LmGU6rMxNbrGK5zPN1i02qzfHxP5si/mB6TrTvO2xeVurO2bNNDROrmpmWrU13vDt6JFmd1SL1Xhe5Vqo2uZCVkvWIkKvRUsd1TfrDX5zWD9mI5nS1UwOCyhH9rBRTIwcJGPCHUxtgAoL4IIBIdjAwAFOtgUaBPSbgyIgeOSbh4h8Plmm3GfQgTRhQ4piFKGRt0nPOq3mEq9Da8EZSLRBb7Ho7xMyC0P4Fd/jDn7BD7iJe8dyRZJDaGnTrMdY5lUHXl0t7Z6Ismnm0NhH9dTMoQb3pVaTtHvSI7IwYnxraXmnNDI1FN3At7hN+t/gGn6iDJzWL+PdJJt63UOPTpm7tObk0SkirsFn3MNvuB3m0eEfKy965mORJapnyozshEEoWcEtXMX1f2bx5F6YnLtLMzuBTcRzGRvKDmGkI4Jx4luUTy51LcKYPGORXQR3qXPiHnlI8XFP+WSNJ2fyhHQIHrESed0ijCpj6pSdOIFO0nN18g//8f0vo9DazRj7BKOwm4QWWEtmG2fZXZVnpd3lpy8b7RGJ8mV/MHgua2NLFofqKd4rFLEjzy06otoHef9W4yZMPbXQzS923/PDxsztQhELxck7udEHyTegH67BdcjTPb8Ho/AYJqBMuazCOmzClrKifFd+KBtxaF8qwVyBA0P5+Rt7z0ot</latexit> <latexit sha1_base64="dHvHwql2rJgZX748PAzRfOjbT2Q=">AAAFMnicnVTPTxNBFH7FVbH+oOjFxEtjg6mGNK9Go+FExEQvRKAUmrDY7C7Tdu3+yu5spWx68mL8Bzh40sSD8eKBE1cT4z+gCZ4wnABvmHDh4JvZjUiBYpymO2/evO/b7715s7pnmQFHXEv1nVJOnznbfy59/sLFSwOZwcszgRv6BisbruX6FV0LmGU6rMxNbrGK5zPN1i02qzfHxP5si/mB6TrTvO2xeVurO2bNNDROrmpmWrU13vDt6JFmd1SL1Xhe5Vqo2uZCVkvWIkKvRUsd1TfrDX5zWD9mI5nS1UwOCyhH9rBRTIwcJGPCHUxtgAoL4IIBIdjAwAFOtgUaBPSbgyIgeOSbh4h8Plmm3GfQgTRhQ4piFKGRt0nPOq3mEq9Da8EZSLRBb7Ho7xMyC0P4Fd/jDn7BD7iJe8dyRZJDaGnTrMdY5lUHXl0t7Z6Ismnm0NhH9dTMoQb3pVaTtHvSI7IwYnxraXmnNDI1FN3At7hN+t/gGn6iDJzWL+PdJJt63UOPTpm7tObk0SkirsFn3MNvuB3m0eEfKy965mORJapnyozshEEoWcEtXMX1f2bx5F6YnLtLMzuBTcRzGRvKDmGkI4Jx4luUTy51LcKYPGORXQR3qXPiHnlI8XFP+WSNJ2fyhHQIHrESed0ijCpj6pSdOIFO0nN18g//8f0vo9DazRj7BKOwm4QWWEtmG2fZXZVnpd3lpy8b7RGJ8mV/MHgua2NLFofqKd4rFLEjzy06otoHef9W4yZMPbXQzS923/PDxsztQhELxck7udEHyTegH67BdcjTPb8Ho/AYJqBMuazCOmzClrKifFd+KBtxaF8qwVyBA0P5+Rt7z0ot</latexit> <latexit sha1_base64="dHvHwql2rJgZX748PAzRfOjbT2Q=">AAAFMnicnVTPTxNBFH7FVbH+oOjFxEtjg6mGNK9Go+FExEQvRKAUmrDY7C7Tdu3+yu5spWx68mL8Bzh40sSD8eKBE1cT4z+gCZ4wnABvmHDh4JvZjUiBYpymO2/evO/b7715s7pnmQFHXEv1nVJOnznbfy59/sLFSwOZwcszgRv6BisbruX6FV0LmGU6rMxNbrGK5zPN1i02qzfHxP5si/mB6TrTvO2xeVurO2bNNDROrmpmWrU13vDt6JFmd1SL1Xhe5Vqo2uZCVkvWIkKvRUsd1TfrDX5zWD9mI5nS1UwOCyhH9rBRTIwcJGPCHUxtgAoL4IIBIdjAwAFOtgUaBPSbgyIgeOSbh4h8Plmm3GfQgTRhQ4piFKGRt0nPOq3mEq9Da8EZSLRBb7Ho7xMyC0P4Fd/jDn7BD7iJe8dyRZJDaGnTrMdY5lUHXl0t7Z6Ismnm0NhH9dTMoQb3pVaTtHvSI7IwYnxraXmnNDI1FN3At7hN+t/gGn6iDJzWL+PdJJt63UOPTpm7tObk0SkirsFn3MNvuB3m0eEfKy965mORJapnyozshEEoWcEtXMX1f2bx5F6YnLtLMzuBTcRzGRvKDmGkI4Jx4luUTy51LcKYPGORXQR3qXPiHnlI8XFP+WSNJ2fyhHQIHrESed0ijCpj6pSdOIFO0nN18g//8f0vo9DazRj7BKOwm4QWWEtmG2fZXZVnpd3lpy8b7RGJ8mV/MHgua2NLFofqKd4rFLEjzy06otoHef9W4yZMPbXQzS923/PDxsztQhELxck7udEHyTegH67BdcjTPb8Ho/AYJqBMuazCOmzClrKifFd+KBtxaF8qwVyBA0P5+Rt7z0ot</latexit> p✓ (x | z) = 1 Z 0 N x | µ✓ (z) , ⌧ 1 Gam (⌧ | a (z) , b (z)) d⌧ = St (x | µ✓ (z) , ✓ (z) , ⌫✓ (z)) <latexit sha1_base64="o3vtbyn0gD2XMGEtLYyDBm6U1kY=">AAAGcXicnVTPaxNBFH6JNtb4o61eFA+GhpZUa5iIohSEYgW9VNvGtoFuG2Y3k2Tt/mIzG5Mue/IiHr304EnBg3jx4MmrIP4DCr0JnrTHCEXowTez22rTJpFuyM6bN+/75ntv3qzqGHqNE7IRix852pc41n88eeLkqdMDg0NnFmq252psXrMN2y2otMYM3WLzXOcGKzguo6ZqsEV1dUqsL9aZW9Nt6yFvOmzZpBVLL+sa5egqDsWeO0Vf4VXGaaAYrMwzikl5VS37jUAx9dLObC1QXL1S5WOjtxTd4jaiSLDio13mzUBGadTw73cg8Tpssks7rnDqrfhXcjsOGeGa/l1q7mAwQrClaCcOtcNCG2UpkFQKphJ58vzwug0sdon2jrN6ckVDsjiYJlkin9R+IxcZaYieGXso9h0UKIENGnhgAgMLONoGUKjhbwlyQMBB3zL46HPR0uU6gwCSiPUwimEERe8qvis4W4q8Fs4FZ02iNdzFwL+LyBSMkC/kDWmRz+Qt+UG2O3L5kkNoaeKohljmFAeenctv9USZOHKo/kV11cyhDDelVh21O9IjstBCfH1tvZWfmBvxR8krson6X5IN8hEzsOq/tNezbO5FFz0qZm7jnKNHxYiwBp/INvlKNr0Msfi7wpOu+RhoierpMiMzYhBK3pOf5AP59t8sjlzzonO3cWQ92EQ8l7Ge7BCGOnyYRr6GfHOpqwFT8oxFdj5cx84Je+QOxoc95aI1HZ3JA9QheMRM5HUJMYqMqWB24gSCqOcq6B/f9R2WUWhtZwx9glHYq4gWWENmG2bZXpVH+a31lafV5oREubI/GDyWtTEli4X1FPsKRezAc/MPqPZe3n/V2BFTVy1483Pt93y/sXA1myPZ3Oy19OTt6BvQDxdgGDJ4z2/AJNyDGZgHLfY7fjGeiY/1tRLnE6nEcBgaj0WYs7DnSVz+A5zbxE8=</latexit> <latexit sha1_base64="o3vtbyn0gD2XMGEtLYyDBm6U1kY=">AAAGcXicnVTPaxNBFH6JNtb4o61eFA+GhpZUa5iIohSEYgW9VNvGtoFuG2Y3k2Tt/mIzG5Mue/IiHr304EnBg3jx4MmrIP4DCr0JnrTHCEXowTez22rTJpFuyM6bN+/75ntv3qzqGHqNE7IRix852pc41n88eeLkqdMDg0NnFmq252psXrMN2y2otMYM3WLzXOcGKzguo6ZqsEV1dUqsL9aZW9Nt6yFvOmzZpBVLL+sa5egqDsWeO0Vf4VXGaaAYrMwzikl5VS37jUAx9dLObC1QXL1S5WOjtxTd4jaiSLDio13mzUBGadTw73cg8Tpssks7rnDqrfhXcjsOGeGa/l1q7mAwQrClaCcOtcNCG2UpkFQKphJ58vzwug0sdon2jrN6ckVDsjiYJlkin9R+IxcZaYieGXso9h0UKIENGnhgAgMLONoGUKjhbwlyQMBB3zL46HPR0uU6gwCSiPUwimEERe8qvis4W4q8Fs4FZ02iNdzFwL+LyBSMkC/kDWmRz+Qt+UG2O3L5kkNoaeKohljmFAeenctv9USZOHKo/kV11cyhDDelVh21O9IjstBCfH1tvZWfmBvxR8krson6X5IN8hEzsOq/tNezbO5FFz0qZm7jnKNHxYiwBp/INvlKNr0Msfi7wpOu+RhoierpMiMzYhBK3pOf5AP59t8sjlzzonO3cWQ92EQ8l7Ge7BCGOnyYRr6GfHOpqwFT8oxFdj5cx84Je+QOxoc95aI1HZ3JA9QheMRM5HUJMYqMqWB24gSCqOcq6B/f9R2WUWhtZwx9glHYq4gWWENmG2bZXpVH+a31lafV5oREubI/GDyWtTEli4X1FPsKRezAc/MPqPZe3n/V2BFTVy1483Pt93y/sXA1myPZ3Oy19OTt6BvQDxdgGDJ4z2/AJNyDGZgHLfY7fjGeiY/1tRLnE6nEcBgaj0WYs7DnSVz+A5zbxE8=</latexit> <latexit sha1_base64="o3vtbyn0gD2XMGEtLYyDBm6U1kY=">AAAGcXicnVTPaxNBFH6JNtb4o61eFA+GhpZUa5iIohSEYgW9VNvGtoFuG2Y3k2Tt/mIzG5Mue/IiHr304EnBg3jx4MmrIP4DCr0JnrTHCEXowTez22rTJpFuyM6bN+/75ntv3qzqGHqNE7IRix852pc41n88eeLkqdMDg0NnFmq252psXrMN2y2otMYM3WLzXOcGKzguo6ZqsEV1dUqsL9aZW9Nt6yFvOmzZpBVLL+sa5egqDsWeO0Vf4VXGaaAYrMwzikl5VS37jUAx9dLObC1QXL1S5WOjtxTd4jaiSLDio13mzUBGadTw73cg8Tpssks7rnDqrfhXcjsOGeGa/l1q7mAwQrClaCcOtcNCG2UpkFQKphJ58vzwug0sdon2jrN6ckVDsjiYJlkin9R+IxcZaYieGXso9h0UKIENGnhgAgMLONoGUKjhbwlyQMBB3zL46HPR0uU6gwCSiPUwimEERe8qvis4W4q8Fs4FZ02iNdzFwL+LyBSMkC/kDWmRz+Qt+UG2O3L5kkNoaeKohljmFAeenctv9USZOHKo/kV11cyhDDelVh21O9IjstBCfH1tvZWfmBvxR8krson6X5IN8hEzsOq/tNezbO5FFz0qZm7jnKNHxYiwBp/INvlKNr0Msfi7wpOu+RhoierpMiMzYhBK3pOf5AP59t8sjlzzonO3cWQ92EQ8l7Ge7BCGOnyYRr6GfHOpqwFT8oxFdj5cx84Je+QOxoc95aI1HZ3JA9QheMRM5HUJMYqMqWB24gSCqOcq6B/f9R2WUWhtZwx9glHYq4gWWENmG2bZXpVH+a31lafV5oREubI/GDyWtTEli4X1FPsKRezAc/MPqPZe3n/V2BFTVy1483Pt93y/sXA1myPZ3Oy19OTt6BvQDxdgGDJ4z2/AJNyDGZgHLfY7fjGeiY/1tRLnE6nEcBgaj0WYs7DnSVz+A5zbxE8=</latexit> <latexit sha1_base64="o3vtbyn0gD2XMGEtLYyDBm6U1kY=">AAAGcXicnVTPaxNBFH6JNtb4o61eFA+GhpZUa5iIohSEYgW9VNvGtoFuG2Y3k2Tt/mIzG5Mue/IiHr304EnBg3jx4MmrIP4DCr0JnrTHCEXowTez22rTJpFuyM6bN+/75ntv3qzqGHqNE7IRix852pc41n88eeLkqdMDg0NnFmq252psXrMN2y2otMYM3WLzXOcGKzguo6ZqsEV1dUqsL9aZW9Nt6yFvOmzZpBVLL+sa5egqDsWeO0Vf4VXGaaAYrMwzikl5VS37jUAx9dLObC1QXL1S5WOjtxTd4jaiSLDio13mzUBGadTw73cg8Tpssks7rnDqrfhXcjsOGeGa/l1q7mAwQrClaCcOtcNCG2UpkFQKphJ58vzwug0sdon2jrN6ckVDsjiYJlkin9R+IxcZaYieGXso9h0UKIENGnhgAgMLONoGUKjhbwlyQMBB3zL46HPR0uU6gwCSiPUwimEERe8qvis4W4q8Fs4FZ02iNdzFwL+LyBSMkC/kDWmRz+Qt+UG2O3L5kkNoaeKohljmFAeenctv9USZOHKo/kV11cyhDDelVh21O9IjstBCfH1tvZWfmBvxR8krson6X5IN8hEzsOq/tNezbO5FFz0qZm7jnKNHxYiwBp/INvlKNr0Msfi7wpOu+RhoierpMiMzYhBK3pOf5AP59t8sjlzzonO3cWQ92EQ8l7Ge7BCGOnyYRr6GfHOpqwFT8oxFdj5cx84Je+QOxoc95aI1HZ3JA9QheMRM5HUJMYqMqWB24gSCqOcq6B/f9R2WUWhtZwx9glHYq4gWWENmG2bZXpVH+a31lafV5oREubI/GDyWtTEli4X1FPsKRezAc/MPqPZe3n/V2BFTVy1483Pt93y/sXA1myPZ3Oy19OTt6BvQDxdgGDJ4z2/AJNyDGZgHLfY7fjGeiY/1tRLnE6nEcBgaj0WYs7DnSVz+A5zbxE8=</latexit> ✓ (z) = a(z)/b (z) , ⌫✓ (z) = 2a (z) <latexit sha1_base64="stBGloYy/piKzEWRtfgWmqUcPS0=">AAAFa3icnVTNaxNBFH+pjdb60dRe/DqEhkgqJb4URSkIxXrwUmwb0wa6NexuJ8ma/WIzG5suOXkRwXMPnhQ8iBcPnrwK4j+g0JvgSYunCL304JvZxbZpk4gTsvPmzfv99vfevFnNNY06R9yKDRwbjB8/MXRy+NTpM2dHEqPnluqO7+msoDum4xU1tc5Mw2YFbnCTFV2PqZZmsmWtNiv2lxvMqxuO/YA3XbZqqRXbKBu6yslVSjxXTApeU0uBwquMqy3FZGWeUSyVV7VysNFSPKNS5RO31X2+iWtal7DJpGL7fbmm1C47pUQKsyhH8rCRi4wURGPeGY19BwXWwAEdfLCAgQ2cbBNUqNNvBXKA4JJvFQLyeWQZcp9BC4YJ61MUowiVvDV6Vmi1EnltWgvOukTr9BaT/h4hk5DGL/gG2/gZ3+IP3O3KFUgOoaVJsxZimVsaeXY+v9MXZdHMobqH6qmZQxluSa0GaXelR2Shh/jGxmY7P72YDq7gK9wm/S9xCz9SBnbjt/56gS2+6KFHo8wdWnPyaBQR1uAT7uJX3PYzaPN3xSc98zHJEtUzZEZWxCCUvMef+AG//TOLK/f86NwdmlkfNhHPZawvO4SRjgDmiG9dPrnUtQ6z8oxFdgHcoM4Je+QuxYc95ZE1F53JfdIheMRK5HWVMIqMqVB24gRaUc9VyD/51/e/jEJrJ2PoE4zCrhFaYE2ZbZhlZ1Ue5Xc2Hz6tNqclypP9weCxrI0lWWyqp3ivUMSOPLfgiGof5N2vxomYemqhm5/rvOeHjaWpbA6zuYXrqZk70TdgCC7BOGTont+EGbgH81CgXNqxC7HxWGrwV3wsfjF+OQwdiEWYMTgw4uk/0+Fb4w==</latexit> <latexit sha1_base64="stBGloYy/piKzEWRtfgWmqUcPS0=">AAAFa3icnVTNaxNBFH+pjdb60dRe/DqEhkgqJb4URSkIxXrwUmwb0wa6NexuJ8ma/WIzG5suOXkRwXMPnhQ8iBcPnrwK4j+g0JvgSYunCL304JvZxbZpk4gTsvPmzfv99vfevFnNNY06R9yKDRwbjB8/MXRy+NTpM2dHEqPnluqO7+msoDum4xU1tc5Mw2YFbnCTFV2PqZZmsmWtNiv2lxvMqxuO/YA3XbZqqRXbKBu6yslVSjxXTApeU0uBwquMqy3FZGWeUSyVV7VysNFSPKNS5RO31X2+iWtal7DJpGL7fbmm1C47pUQKsyhH8rCRi4wURGPeGY19BwXWwAEdfLCAgQ2cbBNUqNNvBXKA4JJvFQLyeWQZcp9BC4YJ61MUowiVvDV6Vmi1EnltWgvOukTr9BaT/h4hk5DGL/gG2/gZ3+IP3O3KFUgOoaVJsxZimVsaeXY+v9MXZdHMobqH6qmZQxluSa0GaXelR2Shh/jGxmY7P72YDq7gK9wm/S9xCz9SBnbjt/56gS2+6KFHo8wdWnPyaBQR1uAT7uJX3PYzaPN3xSc98zHJEtUzZEZWxCCUvMef+AG//TOLK/f86NwdmlkfNhHPZawvO4SRjgDmiG9dPrnUtQ6z8oxFdgHcoM4Je+QuxYc95ZE1F53JfdIheMRK5HWVMIqMqVB24gRaUc9VyD/51/e/jEJrJ2PoE4zCrhFaYE2ZbZhlZ1Ue5Xc2Hz6tNqclypP9weCxrI0lWWyqp3ivUMSOPLfgiGof5N2vxomYemqhm5/rvOeHjaWpbA6zuYXrqZk70TdgCC7BOGTont+EGbgH81CgXNqxC7HxWGrwV3wsfjF+OQwdiEWYMTgw4uk/0+Fb4w==</latexit> <latexit sha1_base64="stBGloYy/piKzEWRtfgWmqUcPS0=">AAAFa3icnVTNaxNBFH+pjdb60dRe/DqEhkgqJb4URSkIxXrwUmwb0wa6NexuJ8ma/WIzG5suOXkRwXMPnhQ8iBcPnrwK4j+g0JvgSYunCL304JvZxbZpk4gTsvPmzfv99vfevFnNNY06R9yKDRwbjB8/MXRy+NTpM2dHEqPnluqO7+msoDum4xU1tc5Mw2YFbnCTFV2PqZZmsmWtNiv2lxvMqxuO/YA3XbZqqRXbKBu6yslVSjxXTApeU0uBwquMqy3FZGWeUSyVV7VysNFSPKNS5RO31X2+iWtal7DJpGL7fbmm1C47pUQKsyhH8rCRi4wURGPeGY19BwXWwAEdfLCAgQ2cbBNUqNNvBXKA4JJvFQLyeWQZcp9BC4YJ61MUowiVvDV6Vmi1EnltWgvOukTr9BaT/h4hk5DGL/gG2/gZ3+IP3O3KFUgOoaVJsxZimVsaeXY+v9MXZdHMobqH6qmZQxluSa0GaXelR2Shh/jGxmY7P72YDq7gK9wm/S9xCz9SBnbjt/56gS2+6KFHo8wdWnPyaBQR1uAT7uJX3PYzaPN3xSc98zHJEtUzZEZWxCCUvMef+AG//TOLK/f86NwdmlkfNhHPZawvO4SRjgDmiG9dPrnUtQ6z8oxFdgHcoM4Je+QuxYc95ZE1F53JfdIheMRK5HWVMIqMqVB24gRaUc9VyD/51/e/jEJrJ2PoE4zCrhFaYE2ZbZhlZ1Ue5Xc2Hz6tNqclypP9weCxrI0lWWyqp3ivUMSOPLfgiGof5N2vxomYemqhm5/rvOeHjaWpbA6zuYXrqZk70TdgCC7BOGTont+EGbgH81CgXNqxC7HxWGrwV3wsfjF+OQwdiEWYMTgw4uk/0+Fb4w==</latexit> <latexit sha1_base64="stBGloYy/piKzEWRtfgWmqUcPS0=">AAAFa3icnVTNaxNBFH+pjdb60dRe/DqEhkgqJb4URSkIxXrwUmwb0wa6NexuJ8ma/WIzG5suOXkRwXMPnhQ8iBcPnrwK4j+g0JvgSYunCL304JvZxbZpk4gTsvPmzfv99vfevFnNNY06R9yKDRwbjB8/MXRy+NTpM2dHEqPnluqO7+msoDum4xU1tc5Mw2YFbnCTFV2PqZZmsmWtNiv2lxvMqxuO/YA3XbZqqRXbKBu6yslVSjxXTApeU0uBwquMqy3FZGWeUSyVV7VysNFSPKNS5RO31X2+iWtal7DJpGL7fbmm1C47pUQKsyhH8rCRi4wURGPeGY19BwXWwAEdfLCAgQ2cbBNUqNNvBXKA4JJvFQLyeWQZcp9BC4YJ61MUowiVvDV6Vmi1EnltWgvOukTr9BaT/h4hk5DGL/gG2/gZ3+IP3O3KFUgOoaVJsxZimVsaeXY+v9MXZdHMobqH6qmZQxluSa0GaXelR2Shh/jGxmY7P72YDq7gK9wm/S9xCz9SBnbjt/56gS2+6KFHo8wdWnPyaBQR1uAT7uJX3PYzaPN3xSc98zHJEtUzZEZWxCCUvMef+AG//TOLK/f86NwdmlkfNhHPZawvO4SRjgDmiG9dPrnUtQ6z8oxFdgHcoM4Je+QuxYc95ZE1F53JfdIheMRK5HWVMIqMqVB24gRaUc9VyD/51/e/jEJrJ2PoE4zCrhFaYE2ZbZhlZ1Ue5Xc2Hz6tNqclypP9weCxrI0lWWyqp3ivUMSOPLfgiGof5N2vxomYemqhm5/rvOeHjaWpbA6zuYXrqZk70TdgCC7BOGTont+EGbgH81CgXNqxC7HxWGrwV3wsfjF+OQwdiEWYMTgw4uk/0+Fb4w==</latexit>
12. 12 Copyright©2018 NTT corp. All Rights Reserved. [Proposed method] Robustness of Student-t decoder °4 °2 0 2 4 x °14 °12 °10 °8 °6 °4 °2 ln (St (x | 0, 1, ∫)) ∫ = 1.0 ∫ = 10.0 ∫ ! 1 • Since the Student-t distribution is heavy-tailed (has large variance), the Student-t decoder is robust to the error between the data point and its decoded mean • The appropriate robustness is set by 𝜈$ 𝐳 , which makes the training of VAE stable! Plot of St(x|0,1, 𝜈) in log scale
13. 13 Copyright©2018 NTT corp. All Rights Reserved. ! "# ! $# 2 ! & [Proposed method] Diagram of Student-t decoder ! "# ! $# ! %# ! & Gaussian decoder Student-t decoder
14. 14 Copyright©2018 NTT corp. All Rights Reserved. • Our approach reduced the negative ELBO equal to or more stably than other approaches [Experiments] Stability of training Negative of Mean ELBO for each dataset 0 200 400 wall clock (seconds) °50 0 50 100 Negative variational lower bound Gaussian MAP(b = 1) MAP(b = 0.001) Student-t 0 50 100 150 200 wall clock (seconds) 0 10 20 Negative variational lower bound Gaussian MAP(b = 1) MAP(b = 0.001) Student-t SMTP Aloi Negative of Mean ELBO Negative of Mean ELBO
15. 15 Copyright©2018 NTT corp. All Rights Reserved. • Our approach obtained the equal to or better density estimation performance than that of other approaches. [Experiments] Test log-likelihoods 1We highlighted the best result in bold, and we also highlighted the results in bold which are not statistically different from the best result according to a pair-wise t-test. Comparison of test log-likelihoods1 Gaussian MAP(b = 1) MAP(b = 0.001) Student-t SMTP -1.248 ± 0.404 -4.864 ± 0.020 -1.932 ± 0.404 0.827 ± 0.105 Aloi 45.418 ± 5.457 -38.210 ± 0.156 30.406 ± 0.383 77.022 ± 0.539 Thyroid 15.519 ± 4.422 -31.266 ± 0.159 18.037 ± 1.318 69.543 ± 0.634 Cancer -18.668 ± 3.448 -45.895 ± 0.843 -19.017 ± 3.273 -18.253 ± 2.629 Satellite -1.852 ± 0.370 -50.895 ± 0.238 -1.899 ± 0.372 -1.811 ± 0.289 <latexit sha1_base64="83E6Yz8E0KLEYfy+XsZtiJGElXo=">AAAIk3icnVXNbxtFFJ+0Bkz4aANCQuplRdwoIDya/fS6CKlpKoFAFmmdtJG6JdpdT5wl+6Xd2dCw2hMXxD/QAyeQOCAuHHriioT4B0DqDYkT9FikXnrom/F6HW9st+pY3n3z3u/95vfezNhO7HspI+T+0pmzjRdefKn58vIrr772+rnzK2/cSKMscemOG/lRsuvYKfW9kO4wj/l0N06oHTg+vekcbvL4zSOapF4UbrPjmN4O7GHo7XuuzcC1t3L2nuXQoRfmzHYy306K3E9gFMuSZLEoTjKfgjkZa9JHdpamnh1Ks8ea1NvYaq07H8rvthbGCSZkFmZN6rNsQEPWbrGZDJbFxQXeoBTX721vjVPbMlY0U2pZcdCSCNaIVuNua9g0tApAFFKLy7irKgsILGc/pYlH05xgU+lUSJnoxVjchh95Y7ymY00eKwJb79QVqSZWZDIh0o2puMpVGFVYNdW5ijodKGgiXle7Ralo++A4ibyBwMs61uVuidKwBhl1RdBFwzihqDsVl01M1HHlMlahvHmKjC7WNbViMlRtrGjTDl2a1PFt4DaMcbtUrGlmIXZNx2ZXr3hMTZ2zYFvuYiJ3KgKloxanVlD0sSQFGwo0SSjq24z6vsdoDY9NfdJTtUMK0SOdTClSVHOuIgB2TxAoxWmALE+YzGrXLCdiLArEObdoOKgu6d75VX59+JBOG3JprKJybEUrS/8gCw1QhFyUoQBRFCIGto9slMLnFpIRQTH4bqMcfAlYnohTVKBlyM0ARQFhg/cQnkOY3Sq9Icw5ZyqyXVjFh28CmRK6SP4kP5GH5A/yM/mXPJ7LlQsOruUY3s4ol8Z75759u//oqVkBvBk6mGQt1MzQPjKFVg+0x8LDq3BH+Udf3X3Yv3T9Yr5GfiAPQP/35D75DSoIj/53f7xGr3+3QI8DlUcwZ+BxADHqwe/kMfmLPMjWSch+2f16YT0+WLx7nqgoKBm4knvkP/Ir+fuZWWIRy8p9j+BNn8LG8UxgM3FCKOjIUQ/47ognE7ruoE2xx7y6HOlwckZn5CrgR2cqAatX7slnoIPz8Bmv6z3IsQRmCNXxHSjKMzcE//uV73kZudY648jHGbl9CNk81xfVjqqsd+WL/qO7n39zcHxJZCXifFD0pehNIFhC6CdflyuiM/ctn9Htad6TaqKSaaEWuPly/Z6fNm4o8H+E5Wva6uUr5W9AE11A76B1uOcddBl9jLbQDnIbFxobjU8anzbfan7QvNK8OoKeWSpz3kRTo9l7Aqmv/pE=</latexit> <latexit sha1_base64="83E6Yz8E0KLEYfy+XsZtiJGElXo=">AAAIk3icnVXNbxtFFJ+0Bkz4aANCQuplRdwoIDya/fS6CKlpKoFAFmmdtJG6JdpdT5wl+6Xd2dCw2hMXxD/QAyeQOCAuHHriioT4B0DqDYkT9FikXnrom/F6HW9st+pY3n3z3u/95vfezNhO7HspI+T+0pmzjRdefKn58vIrr772+rnzK2/cSKMscemOG/lRsuvYKfW9kO4wj/l0N06oHTg+vekcbvL4zSOapF4UbrPjmN4O7GHo7XuuzcC1t3L2nuXQoRfmzHYy306K3E9gFMuSZLEoTjKfgjkZa9JHdpamnh1Ks8ea1NvYaq07H8rvthbGCSZkFmZN6rNsQEPWbrGZDJbFxQXeoBTX721vjVPbMlY0U2pZcdCSCNaIVuNua9g0tApAFFKLy7irKgsILGc/pYlH05xgU+lUSJnoxVjchh95Y7ymY00eKwJb79QVqSZWZDIh0o2puMpVGFVYNdW5ijodKGgiXle7Ralo++A4ibyBwMs61uVuidKwBhl1RdBFwzihqDsVl01M1HHlMlahvHmKjC7WNbViMlRtrGjTDl2a1PFt4DaMcbtUrGlmIXZNx2ZXr3hMTZ2zYFvuYiJ3KgKloxanVlD0sSQFGwo0SSjq24z6vsdoDY9NfdJTtUMK0SOdTClSVHOuIgB2TxAoxWmALE+YzGrXLCdiLArEObdoOKgu6d75VX59+JBOG3JprKJybEUrS/8gCw1QhFyUoQBRFCIGto9slMLnFpIRQTH4bqMcfAlYnohTVKBlyM0ARQFhg/cQnkOY3Sq9Icw5ZyqyXVjFh28CmRK6SP4kP5GH5A/yM/mXPJ7LlQsOruUY3s4ol8Z75759u//oqVkBvBk6mGQt1MzQPjKFVg+0x8LDq3BH+Udf3X3Yv3T9Yr5GfiAPQP/35D75DSoIj/53f7xGr3+3QI8DlUcwZ+BxADHqwe/kMfmLPMjWSch+2f16YT0+WLx7nqgoKBm4knvkP/Ir+fuZWWIRy8p9j+BNn8LG8UxgM3FCKOjIUQ/47ognE7ruoE2xx7y6HOlwckZn5CrgR2cqAatX7slnoIPz8Bmv6z3IsQRmCNXxHSjKMzcE//uV73kZudY648jHGbl9CNk81xfVjqqsd+WL/qO7n39zcHxJZCXifFD0pehNIFhC6CdflyuiM/ctn9Htad6TaqKSaaEWuPly/Z6fNm4o8H+E5Wva6uUr5W9AE11A76B1uOcddBl9jLbQDnIbFxobjU8anzbfan7QvNK8OoKeWSpz3kRTo9l7Aqmv/pE=</latexit> <latexit sha1_base64="83E6Yz8E0KLEYfy+XsZtiJGElXo=">AAAIk3icnVXNbxtFFJ+0Bkz4aANCQuplRdwoIDya/fS6CKlpKoFAFmmdtJG6JdpdT5wl+6Xd2dCw2hMXxD/QAyeQOCAuHHriioT4B0DqDYkT9FikXnrom/F6HW9st+pY3n3z3u/95vfezNhO7HspI+T+0pmzjRdefKn58vIrr772+rnzK2/cSKMscemOG/lRsuvYKfW9kO4wj/l0N06oHTg+vekcbvL4zSOapF4UbrPjmN4O7GHo7XuuzcC1t3L2nuXQoRfmzHYy306K3E9gFMuSZLEoTjKfgjkZa9JHdpamnh1Ks8ea1NvYaq07H8rvthbGCSZkFmZN6rNsQEPWbrGZDJbFxQXeoBTX721vjVPbMlY0U2pZcdCSCNaIVuNua9g0tApAFFKLy7irKgsILGc/pYlH05xgU+lUSJnoxVjchh95Y7ymY00eKwJb79QVqSZWZDIh0o2puMpVGFVYNdW5ijodKGgiXle7Ralo++A4ibyBwMs61uVuidKwBhl1RdBFwzihqDsVl01M1HHlMlahvHmKjC7WNbViMlRtrGjTDl2a1PFt4DaMcbtUrGlmIXZNx2ZXr3hMTZ2zYFvuYiJ3KgKloxanVlD0sSQFGwo0SSjq24z6vsdoDY9NfdJTtUMK0SOdTClSVHOuIgB2TxAoxWmALE+YzGrXLCdiLArEObdoOKgu6d75VX59+JBOG3JprKJybEUrS/8gCw1QhFyUoQBRFCIGto9slMLnFpIRQTH4bqMcfAlYnohTVKBlyM0ARQFhg/cQnkOY3Sq9Icw5ZyqyXVjFh28CmRK6SP4kP5GH5A/yM/mXPJ7LlQsOruUY3s4ol8Z75759u//oqVkBvBk6mGQt1MzQPjKFVg+0x8LDq3BH+Udf3X3Yv3T9Yr5GfiAPQP/35D75DSoIj/53f7xGr3+3QI8DlUcwZ+BxADHqwe/kMfmLPMjWSch+2f16YT0+WLx7nqgoKBm4knvkP/Ir+fuZWWIRy8p9j+BNn8LG8UxgM3FCKOjIUQ/47ognE7ruoE2xx7y6HOlwckZn5CrgR2cqAatX7slnoIPz8Bmv6z3IsQRmCNXxHSjKMzcE//uV73kZudY648jHGbl9CNk81xfVjqqsd+WL/qO7n39zcHxJZCXifFD0pehNIFhC6CdflyuiM/ctn9Htad6TaqKSaaEWuPly/Z6fNm4o8H+E5Wva6uUr5W9AE11A76B1uOcddBl9jLbQDnIbFxobjU8anzbfan7QvNK8OoKeWSpz3kRTo9l7Aqmv/pE=</latexit> <latexit sha1_base64="83E6Yz8E0KLEYfy+XsZtiJGElXo=">AAAIk3icnVXNbxtFFJ+0Bkz4aANCQuplRdwoIDya/fS6CKlpKoFAFmmdtJG6JdpdT5wl+6Xd2dCw2hMXxD/QAyeQOCAuHHriioT4B0DqDYkT9FikXnrom/F6HW9st+pY3n3z3u/95vfezNhO7HspI+T+0pmzjRdefKn58vIrr772+rnzK2/cSKMscemOG/lRsuvYKfW9kO4wj/l0N06oHTg+vekcbvL4zSOapF4UbrPjmN4O7GHo7XuuzcC1t3L2nuXQoRfmzHYy306K3E9gFMuSZLEoTjKfgjkZa9JHdpamnh1Ks8ea1NvYaq07H8rvthbGCSZkFmZN6rNsQEPWbrGZDJbFxQXeoBTX721vjVPbMlY0U2pZcdCSCNaIVuNua9g0tApAFFKLy7irKgsILGc/pYlH05xgU+lUSJnoxVjchh95Y7ymY00eKwJb79QVqSZWZDIh0o2puMpVGFVYNdW5ijodKGgiXle7Ralo++A4ibyBwMs61uVuidKwBhl1RdBFwzihqDsVl01M1HHlMlahvHmKjC7WNbViMlRtrGjTDl2a1PFt4DaMcbtUrGlmIXZNx2ZXr3hMTZ2zYFvuYiJ3KgKloxanVlD0sSQFGwo0SSjq24z6vsdoDY9NfdJTtUMK0SOdTClSVHOuIgB2TxAoxWmALE+YzGrXLCdiLArEObdoOKgu6d75VX59+JBOG3JprKJybEUrS/8gCw1QhFyUoQBRFCIGto9slMLnFpIRQTH4bqMcfAlYnohTVKBlyM0ARQFhg/cQnkOY3Sq9Icw5ZyqyXVjFh28CmRK6SP4kP5GH5A/yM/mXPJ7LlQsOruUY3s4ol8Z75759u//oqVkBvBk6mGQt1MzQPjKFVg+0x8LDq3BH+Udf3X3Yv3T9Yr5GfiAPQP/35D75DSoIj/53f7xGr3+3QI8DlUcwZ+BxADHqwe/kMfmLPMjWSch+2f16YT0+WLx7nqgoKBm4knvkP/Ir+fuZWWIRy8p9j+BNn8LG8UxgM3FCKOjIUQ/47ognE7ruoE2xx7y6HOlwckZn5CrgR2cqAatX7slnoIPz8Bmv6z3IsQRmCNXxHSjKMzcE//uV73kZudY648jHGbl9CNk81xfVjqqsd+WL/qO7n39zcHxJZCXifFD0pehNIFhC6CdflyuiM/ctn9Htad6TaqKSaaEWuPly/Z6fNm4o8H+E5Wva6uUr5W9AE11A76B1uOcddBl9jLbQDnIbFxobjU8anzbfan7QvNK8OoKeWSpz3kRTo9l7Aqmv/pE=</latexit>
16. 16 Copyright©2018 NTT corp. All Rights Reserved. • We proposed the Student-t VAE for robust multivariate density estimation • We experimentally showed that the stability of the training and the high density estimation performance of the Student-t VAE • We recommend using the Student-t distribution as the decoder If you use the VAE for continuous data! In conclusion
17. 17 Copyright©2018 NTT corp. All Rights Reserved. Thank you for your attention! If you have any questions, email me: takahashi.hiroshi@lab.ntt.co.jp Thank you!
18. 18 Copyright©2018 NTT corp. All Rights Reserved. • Q1: Did you compare this model with GAN? • A1: With SMTP dataset, we compared Student-t VAE with Wasserstein GAN, and confirmed that the test log likelihood of the Student-t VAE was better than that of Wasserstein GAN. • Q2: What is the limitation of this approach? • A2: This approach requires heavier computational cost than Gaussian decoder. (about 1.5 times) • Q3: Is this approach useful when the dataset is discrete? • A3: If the dataset is binary, we recommend using the Bernoulli distribution as the decoder. Other than that, our approach may be useful. FAQ