A machine-learning view on heterogeneous catalyst design and discovery

Telluride Workshop on Computational Materials Chemistry, Telluride, Colorado, USA, July 1, 2021.

https://research.chem.ucr.edu/groups/jiang/Telluride_Workshop.html
https://www.telluridescience.org/meetings/workshop-details?wid=901
https://www.telluridescience.org/meetings/workshop-details?wid=945

Transcript

1. / 35 1 A machine-learning view on heterogeneous catalyst design and discovery Ichigaku Takigawa ichigaku.takigawa@riken.jp 1 July 2021 @ Telluride, Colorado Telluride Workshop on Computational Materials Chemistry "EWBODFE
   *OUFMMJHFODF
   1SPKFDU
2. / 35 2 RIKEN Center for AI Project Inst. Chemical Reaction Design & Discovery Hokkaido Univ Two Interrelated Research Interests: Hi, I am a ML researcher working for ML for Stem Cell Biology ML for Chemistry ML with discrete (combinatorial) structures ML for natural sciences Edit-aware graph autocompletion (Hu+) Low-electron dose TEM image improvement (Katsuno+)
3. / 35 3 Today’s talk Gas-phase reactions on sold-phase catalyst surface (Heterogeneous catalysis) https://en.wikipedia.org/wiki/Heterogeneous_catalysis Industrial Synthesis (e.g. Haber-Bosch), Automobile Exhaust Gas Purification, Methane Conversion, etc. Reactants (Gas) Catalysts (Solid) Nano-particle surface Adsorption Diffusion Dissociation Recombination Desorption Our struggles for better ML practices with underspecified, sparse, biased observational data (i.e. a collection of experimental facts from literature)
4. / 35 3 Today’s talk Gas-phase reactions on sold-phase catalyst surface (Heterogeneous catalysis) https://en.wikipedia.org/wiki/Heterogeneous_catalysis Industrial Synthesis (e.g. Haber-Bosch), Automobile Exhaust Gas Purification, Methane Conversion, etc. Reactants (Gas) Catalysts (Solid) Nano-particle surface High Temperature, High Pressure Adsorption Diffusion Dissociation Recombination Desorption Our struggles for better ML practices with underspecified, sparse, biased observational data (i.e. a collection of experimental facts from literature)
5. / 35 4 Today’s talk Gas-phase reactions on sold-phase catalyst surface (Heterogeneous catalysis) Industrial Synthesis (e.g. Haber-Bosch), Automobile Exhaust Gas Purification, Methane Conversion, etc. Reactants (Gas) Catalysts (Solid) Nano-particle surface High Temperature, High Pressure Adsorption Diffusion Dissociation Recombination Desorption God made the bulk; the surface was invented by the devil Devilishly complex too-many-factor process!! —— Wolfgang Pauli Our struggles for better ML practices with underspecified, sparse, biased observational data (i.e. a collection of experimental facts from literature)
6. / 35 5 simulation input output ML: A new way for (lazy) programming computer program input output computer program ML full specification in every detail required give up explicit model instead, grab a tunable model, and show it many input-output instances inductive (empiricism) airhead with a god-like learning capability Random Forest Neural Networks SVR Kernel Ridge All about fitting a very-flexible function to finite points in high-dimensional space. deductive (rationalism)
7. / 35 6 ML: A new way for (lazy) programming ResNet50: 26 million params ResNet101: 45 million params EfficientNet-B7: 66 million params VGG19: 144 million params 12-layer, 12-heads BERT: 110 million params 24-layer, 16-heads BERT: 336 million params GPT-2 XL: 1558 million params GPT-3: 175 billion params simulation input output computer program input output computer program ML full specification in every detail required give up explicit model instead, grab a tunable model, and show it many input-output instances inductive (empiricism) airhead with a god-like learning capability deductive (rationalism) All about fitting a very-flexible function to finite points in high-dimensional space.
8. / 35 6 ML: A new way for (lazy) programming ResNet50: 26 million params ResNet101: 45 million params EfficientNet-B7: 66 million params VGG19: 144 million params 12-layer, 12-heads BERT: 110 million params 24-layer, 16-heads BERT: 336 million params GPT-2 XL: 1558 million params GPT-3: 175 billion params Modern ML: Can we imagine what would happen if we try to fit a function having 175 billion parameters to 100 million data points in 10 thousand dimension?? simulation input output computer program input output computer program ML full specification in every detail required give up explicit model instead, grab a tunable model, and show it many input-output instances inductive (empiricism) airhead with a god-like learning capability deductive (rationalism) All about fitting a very-flexible function to finite points in high-dimensional space.
9. / 35 7 Rashomon Effect: multiplicity of good models ML models are too flexible to overrepresent given finite instances, and many different shapes of functions exist for representing the same finite data. (even if it’s huge) The Rashomon Effect In many practical cases, we have many equally-accurate but different ML models. (the choice of ML methods or the design of NN architectures doesn’t really matter) Note: The Rashomon Effect in ML is attributed to Leo Breiman’s very influential “The Two Cultures” paper published in 2001, but obviously “Rashomon” itself originates from a classic Japanese movie in 1950 by Kurosawa, where four witnesses to a murder describe entirely different contradictory perspectives, but all of them sound true. We often see this in ML competitions. Top ranking solutions are very competitive in performance (equally accurate in practice) but can be very different approaches. 5-CV RMSE: 0.12016 5-CV RMSE: 0.13209 5-CV RMSE: 0.11976 5-CV RMSE: 0.12432 5-CV RMSE: 0.20899 5-CV RMSE: 0.17446 Neural Network Random Forest ExtraTrees GBDT Gaussian Process Kernel Ridge
10. / 35 8 We see differences in underspecified cases right data scarce/underspecified + outliers Neural Networks (ReLU) Random Forest Extra Trees (bootstrap) Neural Networks (ReLU) Random Forest Extra Trees (bootstrap) Neural Networks (Tanh) Linear Regression Kernel Ridge (RBF) Kernel Ridge (Laplacian) Extra Trees (no bootstrap) Gradient Boosting
11. / 35 8 We see differences in underspecified cases right data scarce/underspecified + outliers Neural Networks (ReLU) Random Forest Extra Trees (bootstrap) Neural Networks (ReLU) Random Forest Extra Trees (bootstrap) Neural Networks (Tanh) Linear Regression Kernel Ridge (RBF) Kernel Ridge (Laplacian) Extra Trees (no bootstrap) Gradient Boosting some apparently underfitted?
12. / 35 8 We see differences in underspecified cases right data scarce/underspecified + outliers Neural Networks (ReLU) Random Forest Extra Trees (bootstrap) Neural Networks (ReLU) Random Forest Extra Trees (bootstrap) Neural Networks (Tanh) Linear Regression Kernel Ridge (RBF) Kernel Ridge (Laplacian) Extra Trees (no bootstrap) Gradient Boosting some apparently underfitted? But we often still see the Rashomon (i.e. similar CV performances) and these can predict very differently for further test cases. (in particular, out-of- distribution cases)
13. / 35 9 Designing relevant “inductive biases” Use heuristic assumptions, domain knowledge to constrain/regularize the model space. Prediction Input variables Classifier or Regressor <latexit sha1_base64="Ill3Als4zZd947f5Xm9sW99d0QA=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTomJcEd245CGPBAlp64gNpW3aQkTiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVTU21Hca6PmFsfGJyyj8dmJmdmw+GFhbzttGwFJ5TDM2wirJkc03Vec5RHY0XTYtLdVnjBbm2198vNLllq4Z+4LRMXq5LVV09VhXJISp7WhEroQiLMTfCw0D0QARepIzQIw5xBAMKGqiDQ4dDWIMEm74SRDCYxJXRJs4ipLr7HOcIkLZBWZwyJGJr9K/SquSxOq37NW1XrdApGg2LlGFE2Qu7Zz32zB7YK/v8s1bbrdH30qJZHmi5WQleLGc//lXVaXZw8q0a6dnBMbZdryp5N12mfwtloG+edXrZnUy0vcZu2Rv5v2Fd9kQ30Jvvyl2aZ65H+JHJC70YNUj83Y5hkI/HxK1YPL0RSe56rfJjBatYp34kkMQ+UshR/SoucYWO4BdiwqaQGKQKPk+zhB8hJL8AVA6Qmg==</latexit> x1 <latexit sha1_base64="QFtMwnKe2I12XGZu0bNJbdnDaaE=">AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhAVIwV0caShzwSJGR3HXHDvrK7EJH4Aya2UlhpYmH8AD/Axh+w4BOMJSY2Fl6WTYwS8W5m58yZe+6cmSuZqmI7jHV9wtj4xOSUfzowMzs3HwwtLBZso2HJPC8bqmGVJNHmqqLzvKM4Ki+ZFhc1SeVFqb7X3y82uWUrhn7gtExe0cSarhwrsugQlTutJqqhCIsxN8LDIO6BCLxIG6FHHOIIBmQ0oIFDh0NYhQibvjLiYDCJq6BNnEVIcfc5zhEgbYOyOGWIxNbpX6NV2WN1Wvdr2q5aplNUGhYpw4iyF3bPeuyZPbBX9vlnrbZbo++lRbM00HKzGrxYzn38q9JodnDyrRrp2cExtl2vCnk3XaZ/C3mgb551ermdbLS9xm7ZG/m/YV32RDfQm+/yXYZnr0f4kcgLvRg1KP67HcOgkIjFt2KJzEYkteu1yo8VrGKd+pFECvtII0/1a7jEFTqCX4gJm0JykCr4PM0SfoSQ+gJWLpCb</latexit> x2 <latexit sha1_base64="lFhRrRrVTrFR31ebbMgRp5myJpc=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqcljfKgRCLMCeCwyDqghDcSOqBRxzhGDpkNFAHhwabsAoRFn1FRMFgEFdCmziTkOLsc5zDR9oGZXHKEImt0b9Kq6LLarTu17QctUynqDRMUgYRZi/snvXYM3tgr+zzz1ptp0bfS4tmaaDlRtl/sZz5+FdVp9nGybdqpGcbFew4XhXybjhM/xbyQN886/Qyu+lwe43dsjfyf8O67IluoDXf5bsUT1+P8CORF3oxalD0dzuGQS4WiW5HYqnNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFWE6QnA==</latexit> x3 <latexit sha1_base64="0IPXcU0UIDvzZlYURjV2A/THv9U=">AAACiXichVG7SgNBFD2ur/hM1EawEYNiFWZFNKQKprGMj0TBBNndTHR0X+xOFmLwB6zsRK0ULMQP8ANs/AELP0EsFWwsvNksiAbjXWbnzJl77pyZq7um8CVjz11Kd09vX39sYHBoeGQ0nhgbL/pOzTN4wXBMx9vWNZ+bwuYFKaTJt12Pa5Zu8i39MNfc3wq45wvH3pR1l5ctbc8WVWFokqhiKag40t9NJFmKhTHdDtQIJBFF3knco4QKHBiowQKHDUnYhAafvh2oYHCJK6NBnEdIhPscxxgkbY2yOGVoxB7Sf49WOxFr07pZ0w/VBp1i0vBIOY1Z9sRu2Rt7ZHfshX3+WasR1mh6qdOst7Tc3Y2fTG58/KuyaJbY/1Z19CxRRTr0Ksi7GzLNWxgtfXB09raRWZ9tzLFr9kr+r9gze6Ab2MG7cbPG1y87+NHJC70YNUj93Y52UFxIqUuphbXFZHYlalUMU5jBPPVjGVmsIo8C1T/AKc5xoQwpqpJWMq1UpSvSTOBHKLkvAi+SPA==</latexit> . . . Input representation design, selection, and its engineering (descriptors + feature engineering) Function model vs “Kitchen-sink” models (feeding every possible descriptor) Simple model is enough whenever we can have determining input variables necessary and sufficient to fully define the desired output. Every requirement should be explicitly encoded into model, otherwise we need to provide it through examples. (symmetry, invariance, etc)
14. / 35 10 Designing relevant “inductive biases” Prediction Input variables Function model <latexit sha1_base64="QFtMwnKe2I12XGZu0bNJbdnDaaE=">AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhAVIwV0caShzwSJGR3HXHDvrK7EJH4Aya2UlhpYmH8AD/Axh+w4BOMJSY2Fl6WTYwS8W5m58yZe+6cmSuZqmI7jHV9wtj4xOSUfzowMzs3HwwtLBZso2HJPC8bqmGVJNHmqqLzvKM4Ki+ZFhc1SeVFqb7X3y82uWUrhn7gtExe0cSarhwrsugQlTutJqqhCIsxN8LDIO6BCLxIG6FHHOIIBmQ0oIFDh0NYhQibvjLiYDCJq6BNnEVIcfc5zhEgbYOyOGWIxNbpX6NV2WN1Wvdr2q5aplNUGhYpw4iyF3bPeuyZPbBX9vlnrbZbo++lRbM00HKzGrxYzn38q9JodnDyrRrp2cExtl2vCnk3XaZ/C3mgb551ermdbLS9xm7ZG/m/YV32RDfQm+/yXYZnr0f4kcgLvRg1KP67HcOgkIjFt2KJzEYkteu1yo8VrGKd+pFECvtII0/1a7jEFTqCX4gJm0JykCr4PM0SfoSQ+gJWLpCb</latexit> x2 <latexit sha1_base64="lFhRrRrVTrFR31ebbMgRp5myJpc=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqcljfKgRCLMCeCwyDqghDcSOqBRxzhGDpkNFAHhwabsAoRFn1FRMFgEFdCmziTkOLsc5zDR9oGZXHKEImt0b9Kq6LLarTu17QctUynqDRMUgYRZi/snvXYM3tgr+zzz1ptp0bfS4tmaaDlRtl/sZz5+FdVp9nGybdqpGcbFew4XhXybjhM/xbyQN886/Qyu+lwe43dsjfyf8O67IluoDXf5bsUT1+P8CORF3oxalD0dzuGQS4WiW5HYqnNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFWE6QnA==</latexit> x3 <latexit sha1_base64="0IPXcU0UIDvzZlYURjV2A/THv9U=">AAACiXichVG7SgNBFD2ur/hM1EawEYNiFWZFNKQKprGMj0TBBNndTHR0X+xOFmLwB6zsRK0ULMQP8ANs/AELP0EsFWwsvNksiAbjXWbnzJl77pyZq7um8CVjz11Kd09vX39sYHBoeGQ0nhgbL/pOzTN4wXBMx9vWNZ+bwuYFKaTJt12Pa5Zu8i39MNfc3wq45wvH3pR1l5ctbc8WVWFokqhiKag40t9NJFmKhTHdDtQIJBFF3knco4QKHBiowQKHDUnYhAafvh2oYHCJK6NBnEdIhPscxxgkbY2yOGVoxB7Sf49WOxFr07pZ0w/VBp1i0vBIOY1Z9sRu2Rt7ZHfshX3+WasR1mh6qdOst7Tc3Y2fTG58/KuyaJbY/1Z19CxRRTr0Ksi7GzLNWxgtfXB09raRWZ9tzLFr9kr+r9gze6Ab2MG7cbPG1y87+NHJC70YNUj93Y52UFxIqUuphbXFZHYlalUMU5jBPPVjGVmsIo8C1T/AKc5xoQwpqpJWMq1UpSvSTOBHKLkvAi+SPA==</latexit> . . . Latent variables Learnable variable transformation Representation learning Classifier or Regressor <latexit sha1_base64="Ill3Als4zZd947f5Xm9sW99d0QA=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTomJcEd245CGPBAlp64gNpW3aQkTiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVTU21Hca6PmFsfGJyyj8dmJmdmw+GFhbzttGwFJ5TDM2wirJkc03Vec5RHY0XTYtLdVnjBbm2198vNLllq4Z+4LRMXq5LVV09VhXJISp7WhEroQiLMTfCw0D0QARepIzQIw5xBAMKGqiDQ4dDWIMEm74SRDCYxJXRJs4ipLr7HOcIkLZBWZwyJGJr9K/SquSxOq37NW1XrdApGg2LlGFE2Qu7Zz32zB7YK/v8s1bbrdH30qJZHmi5WQleLGc//lXVaXZw8q0a6dnBMbZdryp5N12mfwtloG+edXrZnUy0vcZu2Rv5v2Fd9kQ30Jvvyl2aZ65H+JHJC70YNUj83Y5hkI/HxK1YPL0RSe56rfJjBatYp34kkMQ+UshR/SoucYWO4BdiwqaQGKQKPk+zhB8hJL8AVA6Qmg==</latexit> x1 Architecture design when we go for representation learning “Kitchen-sink” (raw) inputs Use heuristic assumptions, domain knowledge to constrain/regularize the model space.
15. / 35 10 Designing relevant “inductive biases” Prediction Input variables Function model <latexit sha1_base64="QFtMwnKe2I12XGZu0bNJbdnDaaE=">AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhAVIwV0caShzwSJGR3HXHDvrK7EJH4Aya2UlhpYmH8AD/Axh+w4BOMJSY2Fl6WTYwS8W5m58yZe+6cmSuZqmI7jHV9wtj4xOSUfzowMzs3HwwtLBZso2HJPC8bqmGVJNHmqqLzvKM4Ki+ZFhc1SeVFqb7X3y82uWUrhn7gtExe0cSarhwrsugQlTutJqqhCIsxN8LDIO6BCLxIG6FHHOIIBmQ0oIFDh0NYhQibvjLiYDCJq6BNnEVIcfc5zhEgbYOyOGWIxNbpX6NV2WN1Wvdr2q5aplNUGhYpw4iyF3bPeuyZPbBX9vlnrbZbo++lRbM00HKzGrxYzn38q9JodnDyrRrp2cExtl2vCnk3XaZ/C3mgb551ermdbLS9xm7ZG/m/YV32RDfQm+/yXYZnr0f4kcgLvRg1KP67HcOgkIjFt2KJzEYkteu1yo8VrGKd+pFECvtII0/1a7jEFTqCX4gJm0JykCr4PM0SfoSQ+gJWLpCb</latexit> x2 <latexit sha1_base64="lFhRrRrVTrFR31ebbMgRp5myJpc=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqcljfKgRCLMCeCwyDqghDcSOqBRxzhGDpkNFAHhwabsAoRFn1FRMFgEFdCmziTkOLsc5zDR9oGZXHKEImt0b9Kq6LLarTu17QctUynqDRMUgYRZi/snvXYM3tgr+zzz1ptp0bfS4tmaaDlRtl/sZz5+FdVp9nGybdqpGcbFew4XhXybjhM/xbyQN886/Qyu+lwe43dsjfyf8O67IluoDXf5bsUT1+P8CORF3oxalD0dzuGQS4WiW5HYqnNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFWE6QnA==</latexit> x3 <latexit sha1_base64="0IPXcU0UIDvzZlYURjV2A/THv9U=">AAACiXichVG7SgNBFD2ur/hM1EawEYNiFWZFNKQKprGMj0TBBNndTHR0X+xOFmLwB6zsRK0ULMQP8ANs/AELP0EsFWwsvNksiAbjXWbnzJl77pyZq7um8CVjz11Kd09vX39sYHBoeGQ0nhgbL/pOzTN4wXBMx9vWNZ+bwuYFKaTJt12Pa5Zu8i39MNfc3wq45wvH3pR1l5ctbc8WVWFokqhiKag40t9NJFmKhTHdDtQIJBFF3knco4QKHBiowQKHDUnYhAafvh2oYHCJK6NBnEdIhPscxxgkbY2yOGVoxB7Sf49WOxFr07pZ0w/VBp1i0vBIOY1Z9sRu2Rt7ZHfshX3+WasR1mh6qdOst7Tc3Y2fTG58/KuyaJbY/1Z19CxRRTr0Ksi7GzLNWxgtfXB09raRWZ9tzLFr9kr+r9gze6Ab2MG7cbPG1y87+NHJC70YNUj93Y52UFxIqUuphbXFZHYlalUMU5jBPPVjGVmsIo8C1T/AKc5xoQwpqpJWMq1UpSvSTOBHKLkvAi+SPA==</latexit> . . . Latent variables Learnable variable transformation Representation learning Classifier or Regressor <latexit sha1_base64="Ill3Als4zZd947f5Xm9sW99d0QA=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTomJcEd245CGPBAlp64gNpW3aQkTiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVTU21Hca6PmFsfGJyyj8dmJmdmw+GFhbzttGwFJ5TDM2wirJkc03Vec5RHY0XTYtLdVnjBbm2198vNLllq4Z+4LRMXq5LVV09VhXJISp7WhEroQiLMTfCw0D0QARepIzQIw5xBAMKGqiDQ4dDWIMEm74SRDCYxJXRJs4ipLr7HOcIkLZBWZwyJGJr9K/SquSxOq37NW1XrdApGg2LlGFE2Qu7Zz32zB7YK/v8s1bbrdH30qJZHmi5WQleLGc//lXVaXZw8q0a6dnBMbZdryp5N12mfwtloG+edXrZnUy0vcZu2Rv5v2Fd9kQ30Jvvyl2aZ65H+JHJC70YNUj83Y5hkI/HxK1YPL0RSe56rfJjBatYp34kkMQ+UshR/SoucYWO4BdiwqaQGKQKPk+zhB8hJL8AVA6Qmg==</latexit> x1 Architecture design when we go for representation learning “Kitchen-sink” (raw) inputs Use heuristic assumptions, domain knowledge to constrain/regularize the model space.
16. / 35 10 Designing relevant “inductive biases” Prediction Input variables Function model <latexit sha1_base64="QFtMwnKe2I12XGZu0bNJbdnDaaE=">AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhAVIwV0caShzwSJGR3HXHDvrK7EJH4Aya2UlhpYmH8AD/Axh+w4BOMJSY2Fl6WTYwS8W5m58yZe+6cmSuZqmI7jHV9wtj4xOSUfzowMzs3HwwtLBZso2HJPC8bqmGVJNHmqqLzvKM4Ki+ZFhc1SeVFqb7X3y82uWUrhn7gtExe0cSarhwrsugQlTutJqqhCIsxN8LDIO6BCLxIG6FHHOIIBmQ0oIFDh0NYhQibvjLiYDCJq6BNnEVIcfc5zhEgbYOyOGWIxNbpX6NV2WN1Wvdr2q5aplNUGhYpw4iyF3bPeuyZPbBX9vlnrbZbo++lRbM00HKzGrxYzn38q9JodnDyrRrp2cExtl2vCnk3XaZ/C3mgb551ermdbLS9xm7ZG/m/YV32RDfQm+/yXYZnr0f4kcgLvRg1KP67HcOgkIjFt2KJzEYkteu1yo8VrGKd+pFECvtII0/1a7jEFTqCX4gJm0JykCr4PM0SfoSQ+gJWLpCb</latexit> x2 <latexit sha1_base64="lFhRrRrVTrFR31ebbMgRp5myJpc=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqcljfKgRCLMCeCwyDqghDcSOqBRxzhGDpkNFAHhwabsAoRFn1FRMFgEFdCmziTkOLsc5zDR9oGZXHKEImt0b9Kq6LLarTu17QctUynqDRMUgYRZi/snvXYM3tgr+zzz1ptp0bfS4tmaaDlRtl/sZz5+FdVp9nGybdqpGcbFew4XhXybjhM/xbyQN886/Qyu+lwe43dsjfyf8O67IluoDXf5bsUT1+P8CORF3oxalD0dzuGQS4WiW5HYqnNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFWE6QnA==</latexit> x3 <latexit sha1_base64="0IPXcU0UIDvzZlYURjV2A/THv9U=">AAACiXichVG7SgNBFD2ur/hM1EawEYNiFWZFNKQKprGMj0TBBNndTHR0X+xOFmLwB6zsRK0ULMQP8ANs/AELP0EsFWwsvNksiAbjXWbnzJl77pyZq7um8CVjz11Kd09vX39sYHBoeGQ0nhgbL/pOzTN4wXBMx9vWNZ+bwuYFKaTJt12Pa5Zu8i39MNfc3wq45wvH3pR1l5ctbc8WVWFokqhiKag40t9NJFmKhTHdDtQIJBFF3knco4QKHBiowQKHDUnYhAafvh2oYHCJK6NBnEdIhPscxxgkbY2yOGVoxB7Sf49WOxFr07pZ0w/VBp1i0vBIOY1Z9sRu2Rt7ZHfshX3+WasR1mh6qdOst7Tc3Y2fTG58/KuyaJbY/1Z19CxRRTr0Ksi7GzLNWxgtfXB09raRWZ9tzLFr9kr+r9gze6Ab2MG7cbPG1y87+NHJC70YNUj93Y52UFxIqUuphbXFZHYlalUMU5jBPPVjGVmsIo8C1T/AKc5xoQwpqpJWMq1UpSvSTOBHKLkvAi+SPA==</latexit> . . . Latent variables Learnable variable transformation Representation learning Classifier or Regressor <latexit sha1_base64="Ill3Als4zZd947f5Xm9sW99d0QA=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTomJcEd245CGPBAlp64gNpW3aQkTiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVTU21Hca6PmFsfGJyyj8dmJmdmw+GFhbzttGwFJ5TDM2wirJkc03Vec5RHY0XTYtLdVnjBbm2198vNLllq4Z+4LRMXq5LVV09VhXJISp7WhEroQiLMTfCw0D0QARepIzQIw5xBAMKGqiDQ4dDWIMEm74SRDCYxJXRJs4ipLr7HOcIkLZBWZwyJGJr9K/SquSxOq37NW1XrdApGg2LlGFE2Qu7Zz32zB7YK/v8s1bbrdH30qJZHmi5WQleLGc//lXVaXZw8q0a6dnBMbZdryp5N12mfwtloG+edXrZnUy0vcZu2Rv5v2Fd9kQ30Jvvyl2aZ65H+JHJC70YNUj83Y5hkI/HxK1YPL0RSe56rfJjBatYp34kkMQ+UshR/SoucYWO4BdiwqaQGKQKPk+zhB8hJL8AVA6Qmg==</latexit> x1 Architecture design when we go for representation learning “Kitchen-sink” (raw) inputs Use heuristic assumptions, domain knowledge to constrain/regularize the model space.
17. / 35 10 Designing relevant “inductive biases” Prediction Input variables Function model <latexit sha1_base64="QFtMwnKe2I12XGZu0bNJbdnDaaE=">AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhAVIwV0caShzwSJGR3HXHDvrK7EJH4Aya2UlhpYmH8AD/Axh+w4BOMJSY2Fl6WTYwS8W5m58yZe+6cmSuZqmI7jHV9wtj4xOSUfzowMzs3HwwtLBZso2HJPC8bqmGVJNHmqqLzvKM4Ki+ZFhc1SeVFqb7X3y82uWUrhn7gtExe0cSarhwrsugQlTutJqqhCIsxN8LDIO6BCLxIG6FHHOIIBmQ0oIFDh0NYhQibvjLiYDCJq6BNnEVIcfc5zhEgbYOyOGWIxNbpX6NV2WN1Wvdr2q5aplNUGhYpw4iyF3bPeuyZPbBX9vlnrbZbo++lRbM00HKzGrxYzn38q9JodnDyrRrp2cExtl2vCnk3XaZ/C3mgb551ermdbLS9xm7ZG/m/YV32RDfQm+/yXYZnr0f4kcgLvRg1KP67HcOgkIjFt2KJzEYkteu1yo8VrGKd+pFECvtII0/1a7jEFTqCX4gJm0JykCr4PM0SfoSQ+gJWLpCb</latexit> x2 <latexit sha1_base64="lFhRrRrVTrFR31ebbMgRp5myJpc=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqcljfKgRCLMCeCwyDqghDcSOqBRxzhGDpkNFAHhwabsAoRFn1FRMFgEFdCmziTkOLsc5zDR9oGZXHKEImt0b9Kq6LLarTu17QctUynqDRMUgYRZi/snvXYM3tgr+zzz1ptp0bfS4tmaaDlRtl/sZz5+FdVp9nGybdqpGcbFew4XhXybjhM/xbyQN886/Qyu+lwe43dsjfyf8O67IluoDXf5bsUT1+P8CORF3oxalD0dzuGQS4WiW5HYqnNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFWE6QnA==</latexit> x3 <latexit sha1_base64="0IPXcU0UIDvzZlYURjV2A/THv9U=">AAACiXichVG7SgNBFD2ur/hM1EawEYNiFWZFNKQKprGMj0TBBNndTHR0X+xOFmLwB6zsRK0ULMQP8ANs/AELP0EsFWwsvNksiAbjXWbnzJl77pyZq7um8CVjz11Kd09vX39sYHBoeGQ0nhgbL/pOzTN4wXBMx9vWNZ+bwuYFKaTJt12Pa5Zu8i39MNfc3wq45wvH3pR1l5ctbc8WVWFokqhiKag40t9NJFmKhTHdDtQIJBFF3knco4QKHBiowQKHDUnYhAafvh2oYHCJK6NBnEdIhPscxxgkbY2yOGVoxB7Sf49WOxFr07pZ0w/VBp1i0vBIOY1Z9sRu2Rt7ZHfshX3+WasR1mh6qdOst7Tc3Y2fTG58/KuyaJbY/1Z19CxRRTr0Ksi7GzLNWxgtfXB09raRWZ9tzLFr9kr+r9gze6Ab2MG7cbPG1y87+NHJC70YNUj93Y52UFxIqUuphbXFZHYlalUMU5jBPPVjGVmsIo8C1T/AKc5xoQwpqpJWMq1UpSvSTOBHKLkvAi+SPA==</latexit> . . . Latent variables Learnable variable transformation Representation learning Classifier or Regressor <latexit sha1_base64="Ill3Als4zZd947f5Xm9sW99d0QA=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTomJcEd245CGPBAlp64gNpW3aQkTiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVTU21Hca6PmFsfGJyyj8dmJmdmw+GFhbzttGwFJ5TDM2wirJkc03Vec5RHY0XTYtLdVnjBbm2198vNLllq4Z+4LRMXq5LVV09VhXJISp7WhEroQiLMTfCw0D0QARepIzQIw5xBAMKGqiDQ4dDWIMEm74SRDCYxJXRJs4ipLr7HOcIkLZBWZwyJGJr9K/SquSxOq37NW1XrdApGg2LlGFE2Qu7Zz32zB7YK/v8s1bbrdH30qJZHmi5WQleLGc//lXVaXZw8q0a6dnBMbZdryp5N12mfwtloG+edXrZnUy0vcZu2Rv5v2Fd9kQ30Jvvyl2aZ65H+JHJC70YNUj83Y5hkI/HxK1YPL0RSe56rfJjBatYp34kkMQ+UshR/SoucYWO4BdiwqaQGKQKPk+zhB8hJL8AVA6Qmg==</latexit> x1 Architecture design when we go for representation learning “Kitchen-sink” (raw) inputs Use heuristic assumptions, domain knowledge to constrain/regularize the model space.
18. / 35 10 Designing relevant “inductive biases” Prediction Input variables Function model <latexit sha1_base64="QFtMwnKe2I12XGZu0bNJbdnDaaE=">AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhAVIwV0caShzwSJGR3HXHDvrK7EJH4Aya2UlhpYmH8AD/Axh+w4BOMJSY2Fl6WTYwS8W5m58yZe+6cmSuZqmI7jHV9wtj4xOSUfzowMzs3HwwtLBZso2HJPC8bqmGVJNHmqqLzvKM4Ki+ZFhc1SeVFqb7X3y82uWUrhn7gtExe0cSarhwrsugQlTutJqqhCIsxN8LDIO6BCLxIG6FHHOIIBmQ0oIFDh0NYhQibvjLiYDCJq6BNnEVIcfc5zhEgbYOyOGWIxNbpX6NV2WN1Wvdr2q5aplNUGhYpw4iyF3bPeuyZPbBX9vlnrbZbo++lRbM00HKzGrxYzn38q9JodnDyrRrp2cExtl2vCnk3XaZ/C3mgb551ermdbLS9xm7ZG/m/YV32RDfQm+/yXYZnr0f4kcgLvRg1KP67HcOgkIjFt2KJzEYkteu1yo8VrGKd+pFECvtII0/1a7jEFTqCX4gJm0JykCr4PM0SfoSQ+gJWLpCb</latexit> x2 <latexit sha1_base64="lFhRrRrVTrFR31ebbMgRp5myJpc=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqcljfKgRCLMCeCwyDqghDcSOqBRxzhGDpkNFAHhwabsAoRFn1FRMFgEFdCmziTkOLsc5zDR9oGZXHKEImt0b9Kq6LLarTu17QctUynqDRMUgYRZi/snvXYM3tgr+zzz1ptp0bfS4tmaaDlRtl/sZz5+FdVp9nGybdqpGcbFew4XhXybjhM/xbyQN886/Qyu+lwe43dsjfyf8O67IluoDXf5bsUT1+P8CORF3oxalD0dzuGQS4WiW5HYqnNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFWE6QnA==</latexit> x3 <latexit sha1_base64="0IPXcU0UIDvzZlYURjV2A/THv9U=">AAACiXichVG7SgNBFD2ur/hM1EawEYNiFWZFNKQKprGMj0TBBNndTHR0X+xOFmLwB6zsRK0ULMQP8ANs/AELP0EsFWwsvNksiAbjXWbnzJl77pyZq7um8CVjz11Kd09vX39sYHBoeGQ0nhgbL/pOzTN4wXBMx9vWNZ+bwuYFKaTJt12Pa5Zu8i39MNfc3wq45wvH3pR1l5ctbc8WVWFokqhiKag40t9NJFmKhTHdDtQIJBFF3knco4QKHBiowQKHDUnYhAafvh2oYHCJK6NBnEdIhPscxxgkbY2yOGVoxB7Sf49WOxFr07pZ0w/VBp1i0vBIOY1Z9sRu2Rt7ZHfshX3+WasR1mh6qdOst7Tc3Y2fTG58/KuyaJbY/1Z19CxRRTr0Ksi7GzLNWxgtfXB09raRWZ9tzLFr9kr+r9gze6Ab2MG7cbPG1y87+NHJC70YNUj93Y52UFxIqUuphbXFZHYlalUMU5jBPPVjGVmsIo8C1T/AKc5xoQwpqpJWMq1UpSvSTOBHKLkvAi+SPA==</latexit> . . . Latent variables Learnable variable transformation Representation learning Classifier or Regressor <latexit sha1_base64="Ill3Als4zZd947f5Xm9sW99d0QA=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTomJcEd245CGPBAlp64gNpW3aQkTiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVTU21Hca6PmFsfGJyyj8dmJmdmw+GFhbzttGwFJ5TDM2wirJkc03Vec5RHY0XTYtLdVnjBbm2198vNLllq4Z+4LRMXq5LVV09VhXJISp7WhEroQiLMTfCw0D0QARepIzQIw5xBAMKGqiDQ4dDWIMEm74SRDCYxJXRJs4ipLr7HOcIkLZBWZwyJGJr9K/SquSxOq37NW1XrdApGg2LlGFE2Qu7Zz32zB7YK/v8s1bbrdH30qJZHmi5WQleLGc//lXVaXZw8q0a6dnBMbZdryp5N12mfwtloG+edXrZnUy0vcZu2Rv5v2Fd9kQ30Jvvyl2aZ65H+JHJC70YNUj83Y5hkI/HxK1YPL0RSe56rfJjBatYp34kkMQ+UshR/SoucYWO4BdiwqaQGKQKPk+zhB8hJL8AVA6Qmg==</latexit> x1 Architecture design when we go for representation learning “Kitchen-sink” (raw) inputs Use heuristic assumptions, domain knowledge to constrain/regularize the model space.
19. / 35 10 Designing relevant “inductive biases” Prediction Input variables Function model <latexit sha1_base64="QFtMwnKe2I12XGZu0bNJbdnDaaE=">AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhAVIwV0caShzwSJGR3HXHDvrK7EJH4Aya2UlhpYmH8AD/Axh+w4BOMJSY2Fl6WTYwS8W5m58yZe+6cmSuZqmI7jHV9wtj4xOSUfzowMzs3HwwtLBZso2HJPC8bqmGVJNHmqqLzvKM4Ki+ZFhc1SeVFqb7X3y82uWUrhn7gtExe0cSarhwrsugQlTutJqqhCIsxN8LDIO6BCLxIG6FHHOIIBmQ0oIFDh0NYhQibvjLiYDCJq6BNnEVIcfc5zhEgbYOyOGWIxNbpX6NV2WN1Wvdr2q5aplNUGhYpw4iyF3bPeuyZPbBX9vlnrbZbo++lRbM00HKzGrxYzn38q9JodnDyrRrp2cExtl2vCnk3XaZ/C3mgb551ermdbLS9xm7ZG/m/YV32RDfQm+/yXYZnr0f4kcgLvRg1KP67HcOgkIjFt2KJzEYkteu1yo8VrGKd+pFECvtII0/1a7jEFTqCX4gJm0JykCr4PM0SfoSQ+gJWLpCb</latexit> x2 <latexit sha1_base64="lFhRrRrVTrFR31ebbMgRp5myJpc=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqcljfKgRCLMCeCwyDqghDcSOqBRxzhGDpkNFAHhwabsAoRFn1FRMFgEFdCmziTkOLsc5zDR9oGZXHKEImt0b9Kq6LLarTu17QctUynqDRMUgYRZi/snvXYM3tgr+zzz1ptp0bfS4tmaaDlRtl/sZz5+FdVp9nGybdqpGcbFew4XhXybjhM/xbyQN886/Qyu+lwe43dsjfyf8O67IluoDXf5bsUT1+P8CORF3oxalD0dzuGQS4WiW5HYqnNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFWE6QnA==</latexit> x3 <latexit sha1_base64="0IPXcU0UIDvzZlYURjV2A/THv9U=">AAACiXichVG7SgNBFD2ur/hM1EawEYNiFWZFNKQKprGMj0TBBNndTHR0X+xOFmLwB6zsRK0ULMQP8ANs/AELP0EsFWwsvNksiAbjXWbnzJl77pyZq7um8CVjz11Kd09vX39sYHBoeGQ0nhgbL/pOzTN4wXBMx9vWNZ+bwuYFKaTJt12Pa5Zu8i39MNfc3wq45wvH3pR1l5ctbc8WVWFokqhiKag40t9NJFmKhTHdDtQIJBFF3knco4QKHBiowQKHDUnYhAafvh2oYHCJK6NBnEdIhPscxxgkbY2yOGVoxB7Sf49WOxFr07pZ0w/VBp1i0vBIOY1Z9sRu2Rt7ZHfshX3+WasR1mh6qdOst7Tc3Y2fTG58/KuyaJbY/1Z19CxRRTr0Ksi7GzLNWxgtfXB09raRWZ9tzLFr9kr+r9gze6Ab2MG7cbPG1y87+NHJC70YNUj93Y52UFxIqUuphbXFZHYlalUMU5jBPPVjGVmsIo8C1T/AKc5xoQwpqpJWMq1UpSvSTOBHKLkvAi+SPA==</latexit> . . . Latent variables Learnable variable transformation Representation learning Classifier or Regressor <latexit sha1_base64="Ill3Als4zZd947f5Xm9sW99d0QA=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTomJcEd245CGPBAlp64gNpW3aQkTiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVTU21Hca6PmFsfGJyyj8dmJmdmw+GFhbzttGwFJ5TDM2wirJkc03Vec5RHY0XTYtLdVnjBbm2198vNLllq4Z+4LRMXq5LVV09VhXJISp7WhEroQiLMTfCw0D0QARepIzQIw5xBAMKGqiDQ4dDWIMEm74SRDCYxJXRJs4ipLr7HOcIkLZBWZwyJGJr9K/SquSxOq37NW1XrdApGg2LlGFE2Qu7Zz32zB7YK/v8s1bbrdH30qJZHmi5WQleLGc//lXVaXZw8q0a6dnBMbZdryp5N12mfwtloG+edXrZnUy0vcZu2Rv5v2Fd9kQ30Jvvyl2aZ65H+JHJC70YNUj83Y5hkI/HxK1YPL0RSe56rfJjBatYp34kkMQ+UshR/SoucYWO4BdiwqaQGKQKPk+zhB8hJL8AVA6Qmg==</latexit> x1 Architecture design when we go for representation learning “Kitchen-sink” (raw) inputs Use heuristic assumptions, domain knowledge to constrain/regularize the model space.
20. / 35 10 Designing relevant “inductive biases” Prediction Input variables Function model <latexit sha1_base64="QFtMwnKe2I12XGZu0bNJbdnDaaE=">AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhAVIwV0caShzwSJGR3HXHDvrK7EJH4Aya2UlhpYmH8AD/Axh+w4BOMJSY2Fl6WTYwS8W5m58yZe+6cmSuZqmI7jHV9wtj4xOSUfzowMzs3HwwtLBZso2HJPC8bqmGVJNHmqqLzvKM4Ki+ZFhc1SeVFqb7X3y82uWUrhn7gtExe0cSarhwrsugQlTutJqqhCIsxN8LDIO6BCLxIG6FHHOIIBmQ0oIFDh0NYhQibvjLiYDCJq6BNnEVIcfc5zhEgbYOyOGWIxNbpX6NV2WN1Wvdr2q5aplNUGhYpw4iyF3bPeuyZPbBX9vlnrbZbo++lRbM00HKzGrxYzn38q9JodnDyrRrp2cExtl2vCnk3XaZ/C3mgb551ermdbLS9xm7ZG/m/YV32RDfQm+/yXYZnr0f4kcgLvRg1KP67HcOgkIjFt2KJzEYkteu1yo8VrGKd+pFECvtII0/1a7jEFTqCX4gJm0JykCr4PM0SfoSQ+gJWLpCb</latexit> x2 <latexit sha1_base64="lFhRrRrVTrFR31ebbMgRp5myJpc=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqcljfKgRCLMCeCwyDqghDcSOqBRxzhGDpkNFAHhwabsAoRFn1FRMFgEFdCmziTkOLsc5zDR9oGZXHKEImt0b9Kq6LLarTu17QctUynqDRMUgYRZi/snvXYM3tgr+zzz1ptp0bfS4tmaaDlRtl/sZz5+FdVp9nGybdqpGcbFew4XhXybjhM/xbyQN886/Qyu+lwe43dsjfyf8O67IluoDXf5bsUT1+P8CORF3oxalD0dzuGQS4WiW5HYqnNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFWE6QnA==</latexit> x3 <latexit sha1_base64="0IPXcU0UIDvzZlYURjV2A/THv9U=">AAACiXichVG7SgNBFD2ur/hM1EawEYNiFWZFNKQKprGMj0TBBNndTHR0X+xOFmLwB6zsRK0ULMQP8ANs/AELP0EsFWwsvNksiAbjXWbnzJl77pyZq7um8CVjz11Kd09vX39sYHBoeGQ0nhgbL/pOzTN4wXBMx9vWNZ+bwuYFKaTJt12Pa5Zu8i39MNfc3wq45wvH3pR1l5ctbc8WVWFokqhiKag40t9NJFmKhTHdDtQIJBFF3knco4QKHBiowQKHDUnYhAafvh2oYHCJK6NBnEdIhPscxxgkbY2yOGVoxB7Sf49WOxFr07pZ0w/VBp1i0vBIOY1Z9sRu2Rt7ZHfshX3+WasR1mh6qdOst7Tc3Y2fTG58/KuyaJbY/1Z19CxRRTr0Ksi7GzLNWxgtfXB09raRWZ9tzLFr9kr+r9gze6Ab2MG7cbPG1y87+NHJC70YNUj93Y52UFxIqUuphbXFZHYlalUMU5jBPPVjGVmsIo8C1T/AKc5xoQwpqpJWMq1UpSvSTOBHKLkvAi+SPA==</latexit> . . . Latent variables Learnable variable transformation Representation learning Classifier or Regressor Linear <latexit sha1_base64="Ill3Als4zZd947f5Xm9sW99d0QA=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTomJcEd245CGPBAlp64gNpW3aQkTiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVTU21Hca6PmFsfGJyyj8dmJmdmw+GFhbzttGwFJ5TDM2wirJkc03Vec5RHY0XTYtLdVnjBbm2198vNLllq4Z+4LRMXq5LVV09VhXJISp7WhEroQiLMTfCw0D0QARepIzQIw5xBAMKGqiDQ4dDWIMEm74SRDCYxJXRJs4ipLr7HOcIkLZBWZwyJGJr9K/SquSxOq37NW1XrdApGg2LlGFE2Qu7Zz32zB7YK/v8s1bbrdH30qJZHmi5WQleLGc//lXVaXZw8q0a6dnBMbZdryp5N12mfwtloG+edXrZnUy0vcZu2Rv5v2Fd9kQ30Jvvyl2aZ65H+JHJC70YNUj83Y5hkI/HxK1YPL0RSe56rfJjBatYp34kkMQ+UshR/SoucYWO4BdiwqaQGKQKPk+zhB8hJL8AVA6Qmg==</latexit> x1 Architecture design when we go for representation learning “Kitchen-sink” (raw) inputs Again, simple model is enough when we have good features. Use heuristic assumptions, domain knowledge to constrain/regularize the model space.
21. / 35 11 A toy example: approximate adders Try to teach ML “arithmetic addition” only by examples. We can also add both 5+6 and 6+5 to tell ML by examples that addition is commutative. adder <latexit sha1_base64="tiacEhQgmTkomNeV5LHmrY6hmbk=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTXxhXRDcuecgjQULaOmJDaZu2EJH4AyZuZeFKExfGD/AD3PgDLvgE4xITNy68lCZGiXib6Zw5c8+dM3NlU1Nth7GOTxgZHRuf8E8GpqZnZoOhufmcbdQthWcVQzOsgizZXFN1nnVUR+MF0+JSTdZ4Xq7u9fbzDW7ZqqEfOE2Tl2pSRVePVUVyiMqclsVyKMJizI3wIBA9EIEXSSP0iEMcwYCCOmrg0OEQ1iDBpq8IEQwmcSW0iLMIqe4+xzkCpK1TFqcMidgq/Su0KnqsTuteTdtVK3SKRsMiZRhR9sLuWZc9swf2yj7/rNVya/S8NGmW+1puloMXi5mPf1U1mh2cfKuGenZwjG3Xq0reTZfp3ULp6xtn7W5mJx1trbBb9kb+b1iHPdEN9Ma7cpfi6eshfmTyQi9GDRJ/t2MQ5NZi4lZsPbURSex6rfJjCctYpX7EkcA+kshS/QoucYW24BdiwqYQ76cKPk+zgB8hJL4AVGCQmw==</latexit> x1 <latexit sha1_base64="4Sn0JXQ8Nli9zYaRMiYfd4a9JHg=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqclmPlQIhFmBPBYRB1QQhuJPXAI45wDB0yGqiDQ4NNWIUIi74iomAwiCuhTZxJSHH2Oc7hI22DsjhliMTW6F+lVdFlNVr3a1qOWqZTVBomKYMIsxd2z3rsmT2wV/b5Z622U6PvpUWzNNByo+y/WM58/Kuq02zj5Fs10rONCnYcrwp5Nxymfwt5oG+edXqZ3XS4vcZu2Rv5v2Fd9kQ30Jrv8l2Kp69H+JHIC70YNSj6ux3DIBeLRLcjG6nNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFVoCQnA==</latexit> x2 <latexit sha1_base64="SKy45aVsIvNNIc9wr1Q6tiMY1uI=">AAACinichVHLSsNAFL2Nr1qrrboR3BRLRRDKpIrPTVEXLvuwD6glJHFah+ZFkpbW4g+4cyXYlYIL8QP8ADf+gIt+gris4MaFN2lAtFhvmMyZM/fcOTNXMhRm2YR0fdzI6Nj4hH8yMBWcngmFZ+fyll43ZZqTdUU3i5JoUYVpNGczW6FFw6SiKim0INX2nf1Cg5oW07Uju2XQsipWNVZhsmgjVWgK/GpTSAjhKIkTNyKDgPdAFLxI6eFHOIYT0EGGOqhAQQMbsQIiWPiVgAcCBnJlaCNnImLuPoVzCKC2jlkUM0Rka/iv4qrksRqunZqWq5bxFAWHicoIxMgLuSc98kweyCv5/LNW263heGnhLPW11BBCFwvZj39VKs42nH6rhnq2oQJbrleG3g2XcW4h9/WNs6tedicTay+TW/KG/m9IlzzhDbTGu3yXppnOED8SesEXwwbxv9sxCPKJOL8RX0uvR5N7Xqv8sAhLsIL92IQkHEIKcm79S7iGDhfkEtw2t9tP5XyeZh5+BHfwBX8Zkfc=</latexit> x1 + x2 <latexit sha1_base64="1coS+Mt9eL/r4jEoYDkZ5NYfoeE=">AAACiHichVHLSsNAFD3Gd3206kZwUywVQSg3KlYFQerGZVutCiqSxGkdTJOQpIVa/AE3LlVcKbgQP8APcOMPuPATxKWCGxfepgFRUW+YzJkz99w5M1d3TOn5RI8tSmtbe0dnV3ekp7evPxobGFzz7IpriIJhm7a7oWueMKUlCr70TbHhuEIr66ZY1/eXGvvrVeF60rZW/ZojtstayZJFaWg+UwV1YmpheieWoBQFEf8J1BAkEEbWjt1iC7uwYaCCMgQs+IxNaPD424QKgsPcNurMuYxksC9wiAhrK5wlOENjdp//JV5thqzF60ZNL1AbfIrJw2VlHEl6oGt6oXu6oSd6/7VWPajR8FLjWW9qhbMTPRpeeftXVebZx96n6k/PPoqYDbxK9u4ETOMWRlNfPTh5WZnPJ+tjdEnP7P+CHumOb2BVX42rnMif/+FHZy/8Ytwg9Xs7foK1yZQ6k5rKTScWM2GrujCCUYxzP9JYxDKyKHB9iWOc4kyJKKSklblmqtISaobwJZTMB48DkKc=</latexit> 1 + 3 = 4 <latexit sha1_base64="Jl1LVcl+mtrFJMn2AKjNmRINEM4=">AAACiHichVHLSsNAFD3GV62vqhvBjVgUQSi39VEVhFI3LtVaW1CRJE7rYJqEJC3U4g+4caniSsGF+AF+gBt/wIWfIC4ruHHhbRoQFfWGyZw5c8+dM3M125CuR/TUorS2tXd0hrrC3T29ff2RgcFN1yo7usjqlmE5eU11hSFNkfWkZ4i87Qi1pBkipx0sN/ZzFeG40jI3vKotdkpq0ZQFqaseU9nE1OxScjcSpRj5MfoTxAMQRRCrVuQO29iDBR1llCBgwmNsQIXL3xbiINjM7aDGnMNI+vsCRwiztsxZgjNUZg/4X+TVVsCavG7UdH21zqcYPBxWjmKcHumG6vRAt/RM77/Wqvk1Gl6qPGtNrbB3+4+HM2//qko8e9j/VP3p2UMB875Xyd5tn2ncQm/qK4en9czi+nhtgq7ohf1f0hPd8w3Myqt+vSbWL/7wo7EXfjFuUPx7O36CzUQsPhebXpuJptJBq0IYwRgmuR9JpLCCVWS5vsQJznCuhBVSkspCM1VpCTRD+BJK+gOby5Ct</latexit> 2 + 5 = 7 <latexit sha1_base64="NRtmyWkgZ5nlm1ETxVkZxPMO8Rc=">AAACiXichVHNLgNRGD0df6V+io3ERjREImnuUJREIrqxpNWSIM3MuGV0/jJz24TGC1jZCVYkFuIBPICNF7DwCGJZiY2Fb6aTCKK+yZ177rnf+e6591MdQ/cEY88RqaW1rb0j2tkV6+7p7Yv3DxQ8u+JqPK/Zhu1uqorHDd3ieaELg286LldM1eAbajnj729UuevptrUuDh2+Yyp7ll7SNUUQVZiZnF+UU8V4giVZECO/gRyCBMJYteP32MYubGiowASHBUHYgAKPvi3IYHCI20GNOJeQHuxzHKOLtBXK4pShEFum/x6ttkLWorVf0wvUGp1i0HBJOYIx9sRuWZ09sjv2wj7+rFULavheDmlWG1ruFPtOhnLv/6pMmgX2v1RNPQuUkA686uTdCRj/FlpDXz06q+cWsmO1cXbNXsn/FXtmD3QDq/qm3azx7GUTPyp5oRejBsk/2/EbFKaS8mxyei2VWFoOWxXFMEYxQf2YwxJWsIo81T/AKc5xIcUkWUpLC41UKRJqBvEtpMwnMx2Q7A==</latexit> 5 + 9 = 14 <latexit sha1_base64="wFMJb33bMt0hW8th8BMIqvUdNU8=">AAACinichVFLSwJRFD5OLzNLq03QRhIjCOSMRk8CqRYtfeQDTGRmutrgvJgZBZP+QLtWQa4KWkQ/oB/Qpj/Qwp8QLQ3atOg4DkRJdoY797vfPd+5371HNBTZshE7Hm5kdGx8wjvpm/JPzwSCs3M5S6+bEstKuqKbBVGwmCJrLGvLtsIKhskEVVRYXqzt9/bzDWZasq4d2U2DlVShqskVWRJsovLxVR53+Xg5GMYoOhEaBLwLwuBGUg8+wjGcgA4S1EEFBhrYhBUQwKKvCDwgGMSVoEWcSUh29hmcg4+0dcpilCEQW6N/lVZFl9Vo3atpOWqJTlFomKQMQQRf8B67+IwP+Iqff9ZqOTV6Xpo0i30tM8qBi4XMx78qlWYbTr9VQz3bUIFNx6tM3g2H6d1C6usbZ1fdzHY60lrGW3wj/zfYwSe6gdZ4l+5SLN0e4kckL/Ri1CD+dzsGQS4W5dej8dRaOLHntsoLi7AEK9SPDUjAISQh69S/hGtoc34uxm1xO/1UzuNq5uFHcAdfqESRGw==</latexit> 3 + 10 = 13 <latexit sha1_base64="FoI7hqDrtQtFjlgLPgriDwtw1iU=">AAACiXichVHLSsNAFD2Nr1pfVTeCm2JRBKFMfNQiCKIbl33YVlApSRx1NE1CMi1o8QdcuRN1peBC/AA/wI0/4MJPEJcV3LjwNg2IFvWGyZw5c8+dM3N1xxSeZOw5pLS1d3R2hbsjPb19/QPRwaGCZ1dcg+cN27TddV3zuCksnpdCmnzdcblW1k1e1A9WGvvFKnc9YVtr8tDhW2Vt1xI7wtAkUYW5qeSiqpaicZZgfsRagRqAOIJI29F7bGIbNgxUUAaHBUnYhAaPvg2oYHCI20KNOJeQ8Pc5jhEhbYWyOGVoxB7Qf5dWGwFr0bpR0/PVBp1i0nBJGcM4e2K3rM4e2R17YR+/1qr5NRpeDmnWm1rulAZORnLv/6rKNEvsfan+9Cyxg5TvVZB3x2catzCa+urRWT23kB2vTbBr9kr+r9gze6AbWNU34ybDs5d/+NHJC70YNUj92Y5WUJhOqMnETGY2vrQctCqMUYxhkvoxjyWsIo081d/HKc5xofQoqpJSFpqpSijQDONbKCufJlSQ5g==</latexit> 5 + 6 = 11 <latexit sha1_base64="n3Yy/9dJfdSB5yYDzUvImVPJv/g=">AAACiHichVHLSsNAFD3Gd3206kZwIxZFEMqNii8QRTcu1VotVClJnNahaRKStKDFH3DjUsWVggvxA/wAN/6Ai36CuFRw48LbNCAq1hsmc+bMPXfOzNUdU3o+UbVJaW5pbWvv6Ix0dff0RmN9/dueXXINkTJs03bTuuYJU1oi5UvfFGnHFVpRN8WOXlit7e+UhetJ29ryDx2xV9TylsxJQ/OZSqkT6uJSNhanBAUx/BuoIYgjjHU7do9d7MOGgRKKELDgMzahweMvAxUEh7k9VJhzGclgX+AYEdaWOEtwhsZsgf95XmVC1uJ1raYXqA0+xeThsnIYo/REt/RKj3RHz/TxZ61KUKPm5ZBnva4VTjZ6Mph8/1dV5NnHwZeqoWcfOcwFXiV7dwKmdgujri8fnb0mFzZHK2N0TS/s/4qq9MA3sMpvxs2G2Lxs4EdnL/xi3CD1Zzt+g+3JhDqTmNqYji+vhK3qwBBGMM79mMUy1rCOFNeXOMU5LpSIQsqsMl9PVZpCzQC+hbLyCaIfkLA=</latexit> 1 + 1 =? <latexit sha1_base64="hRdVP+e2y4bjP1zOby3ZyPytH4w=">AAACi3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKRRhqa+gaatraxwsoG+gZgIECJsMQylBmgIKAfIHtDDEMKQz5DMkMpQy5DKkMeQwlQHYOQyJDMRBGMxgyGDAUAMViGaqBYkVAViZYPpWhloELqLcUqCoVqCIRKJoNJNOBvGioaB6QDzKzGKw7GWhLDhAXAXUqMKgaXDVYafDZ4ITBaoOXBn9wmlUNNgPklkognQTRm1oQz98lEfydoK5cIF3CkIHQhdfNJQxpDBZgt2YC3V4AFgH5Ihmiv6xq+udgqyDVajWDRQavge5faHDT4DDQB3llX5KXBqYGzcbjniSgW4AhBowgQ/TowGSEGekZmukZB5ooOzhBo4qDQZpBiUEDGB/mDA4MHgwBDKHgeJjEMJthDhMvkzGTFZMNRCkTI1SPMAMKYHIFACGukUw=</latexit> 1 + ( 1) =? <latexit sha1_base64="iSckyhI7u8UoVI74y32YMTkgc7c=">AAACj3ichVHLSsNAFD3Gd3201Y3gRiwVQSiTVmwrqEU3umutVUFFkjitoWkSkmlBiz/gB+jChQ9wIX6AH+DGH3DhJ4jLCm5ceJsGREW9YTJnztxz58xc1TZ0VzD21Ca1d3R2dff0Bvr6BwaDofDQumtVHY0XNMuwnE1Vcbmhm7wgdGHwTdvhSkU1+IZaXmrub9S44+qWuSYObL5TUUqmXtQ1RRC1LcdT6fhUOpVIzi3shiIsxrwY+wlkH0TgR9YK3WEbe7CgoYoKOEwIwgYUuPRtQQaDTdwO6sQ5hHRvn+MIAdJWKYtThkJsmf4lWm35rEnrZk3XU2t0ikHDIeUYouyR3bAGe2C37Jm9/1qr7tVoejmgWW1pub0bPB7Jv/2rqtAssP+p+tOzQBEpz6tO3m2Pad5Ca+lrh6eN/OxqtD7BrtgL+b9kT+yebmDWXrXrHF89+8OPSl7oxahB8vd2/ATr8Zg8E0vkpiOZRb9VPRjFOCapH0lksIwsClTfxgnOcSGFpaQ0L2VaqVKbrxnGl5BWPgDiqZJ1</latexit> 12892 + 9837 =? <latexit sha1_base64="3cCCvxejuSwhH3gAGZntIRJA/Ck=">AAAChHichVHNLgNRFP46/uunxUZi02iIhTRnWtWOhTRsLFsUCSIz42LS6cxkZtqExguwJRZWJBbiATyAjRew8AhiSWJj4cy0IhY4N/fec79zvnO/e4/mmIbnEz1FpLb2js6u7p5ob1//QCw+OLTq2TVXF2XdNm13XVM9YRqWKPuGb4p1xxVqVTPFmlZZCOJrdeF6hm2t+AeO2Kqqe5axa+iqz1ApvR1PUkrJZ7OZfIJSREpaybGjKIqckxMyI4El0bKiHb/DJnZgQ0cNVQhY8Nk3ocLjsQEZBIexLTQYc9kzwrjAEaLMrXGW4AyV0Qqve3zaaKEWn4OaXsjW+RaTp8vMBMbpkW7olR7olp7p49dajbBGoOWAd63JFc527Hhk+f1fVpV3H/vfrD81+9hFPtRqsHYnRIJX6E1+/fD8dXl2abwxQVf0wvov6Ynu+QVW/U2/Lomliz/0aKyFf4wb9NWFxO/Oajolz6QypelkYb7Vqm6MYgyT3I8cClhEEWWuL3CCU5xJndKUlJGyzVQp0uIM44dJc58RFJAW</latexit> 2 <latexit sha1_base64="dtxzgykiOKrKKToFg2z7LT9XiS0=">AAAChHichVHNLgNRFP6Mv/ovNhKbRkMspDnTqnYsRNhYaikSRGbGbU1MZyYz0ybVeAG2xMKKxEI8gAew8QIWfQSxJLGxcGZaEQucm3vvud8537nfvUdzTMPziRptUntHZ1d3pKe3r39gcCg6PLLh2RVXFwXdNm13S1M9YRqWKPiGb4otxxVqWTPFpna4HMQ3q8L1DNta92uO2C2rJcsoGrrqM5SjvWicEko2nU5lY5QgUpJKhh1FUeSMHJMZCSyOlq3a0XvsYB82dFRQhoAFn30TKjwe25BBcBjbRZ0xlz0jjAsco5e5Fc4SnKEyeshriU/bLdTic1DTC9k632LydJkZwyQ90S290iPd0TN9/FqrHtYItNR415pc4ewNnYytvf/LKvPu4+Cb9admH0VkQ60Ga3dCJHiF3uRXjy5e1+bzk/UpuqYX1n9FDXrgF1jVN/0mJ/KXf+jRWAv/GDfoqwux352NZEKeS6Rys/HFpVarIhjHBKa5HxksYgWrKHB9gVOc4VzqkmaklJRupkptLc4ofpi08AkM1JAU</latexit> 0 <latexit sha1_base64="P9nwiNSxLZKIKI1oebJys0uGCoU=">AAACiHichVHLLgRBFD3ae7wGG4nNxIRYTW63R09bydhYjscgQaS7FSp6ujvdPZMw8QM2logViYX4AB9g4wcsfIJYkthYuN0zIha4laq6de49t07VtXxHhhHRU5PS3NLa1t7Rmerq7untS/cPrIReJbBFyfYcL1izzFA40hWlSEaOWPMDYZYtR6xa+3NxfLUqglB67nJ04IvNsrnryh1pmxFDJU3TNWMrnaWckZ+amshnKEdkaIbOjmEYqq5mVEZiy6JhRS99hw1sw4ONCsoQcBGx78BEyGMdKgg+Y5uoMRawJ5O4wBFSzK1wluAMk9F9Xnf5tN5AXT7HNcOEbfMtDs+AmRmM0iPd0Cs90C0908evtWpJjVjLAe9WnSv8rb7joaX3f1ll3iPsfbP+1BxhB/lEq2TtfoLEr7Dr/Orh6evSzOJobYyu6IX1X9IT3fML3Oqbfb0gFi/+0GOxFv4xbtBXFzK/OytaTp3OTSxMZmcLjVZ1YBgjGOd+6JjFPIoocX2JE5zhXEkppOiKUU9VmhqcQfwwpfAJa7yREg==</latexit> 22729 train test (the case where a clear answer and underlying logic exist)
22. / 35 12 A toy example: approximate adders RF says 1 + 1 = 5.15 and 1 - 1 = 5.15 and 12892 + 9837 = 12.75 MLP better? But anyway it’s totally wrong. 2-10-5-1 Feed Forward NN (MLP, Multi-Layer Perceptron) adder <latexit sha1_base64="tiacEhQgmTkomNeV5LHmrY6hmbk=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTXxhXRDcuecgjQULaOmJDaZu2EJH4AyZuZeFKExfGD/AD3PgDLvgE4xITNy68lCZGiXib6Zw5c8+dM3NlU1Nth7GOTxgZHRuf8E8GpqZnZoOhufmcbdQthWcVQzOsgizZXFN1nnVUR+MF0+JSTdZ4Xq7u9fbzDW7ZqqEfOE2Tl2pSRVePVUVyiMqclsVyKMJizI3wIBA9EIEXSSP0iEMcwYCCOmrg0OEQ1iDBpq8IEQwmcSW0iLMIqe4+xzkCpK1TFqcMidgq/Su0KnqsTuteTdtVK3SKRsMiZRhR9sLuWZc9swf2yj7/rNVya/S8NGmW+1puloMXi5mPf1U1mh2cfKuGenZwjG3Xq0reTZfp3ULp6xtn7W5mJx1trbBb9kb+b1iHPdEN9Ma7cpfi6eshfmTyQi9GDRJ/t2MQ5NZi4lZsPbURSex6rfJjCctYpX7EkcA+kshS/QoucYW24BdiwqYQ76cKPk+zgB8hJL4AVGCQmw==</latexit> x1 <latexit sha1_base64="4Sn0JXQ8Nli9zYaRMiYfd4a9JHg=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqclmPlQIhFmBPBYRB1QQhuJPXAI45wDB0yGqiDQ4NNWIUIi74iomAwiCuhTZxJSHH2Oc7hI22DsjhliMTW6F+lVdFlNVr3a1qOWqZTVBomKYMIsxd2z3rsmT2wV/b5Z622U6PvpUWzNNByo+y/WM58/Kuq02zj5Fs10rONCnYcrwp5Nxymfwt5oG+edXqZ3XS4vcZu2Rv5v2Fd9kQ30Jrv8l2Kp69H+JHIC70YNSj6ux3DIBeLRLcjG6nNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFVoCQnA==</latexit> x2 <latexit sha1_base64="SKy45aVsIvNNIc9wr1Q6tiMY1uI=">AAACinichVHLSsNAFL2Nr1qrrboR3BRLRRDKpIrPTVEXLvuwD6glJHFah+ZFkpbW4g+4cyXYlYIL8QP8ADf+gIt+gris4MaFN2lAtFhvmMyZM/fcOTNXMhRm2YR0fdzI6Nj4hH8yMBWcngmFZ+fyll43ZZqTdUU3i5JoUYVpNGczW6FFw6SiKim0INX2nf1Cg5oW07Uju2XQsipWNVZhsmgjVWgK/GpTSAjhKIkTNyKDgPdAFLxI6eFHOIYT0EGGOqhAQQMbsQIiWPiVgAcCBnJlaCNnImLuPoVzCKC2jlkUM0Rka/iv4qrksRqunZqWq5bxFAWHicoIxMgLuSc98kweyCv5/LNW263heGnhLPW11BBCFwvZj39VKs42nH6rhnq2oQJbrleG3g2XcW4h9/WNs6tedicTay+TW/KG/m9IlzzhDbTGu3yXppnOED8SesEXwwbxv9sxCPKJOL8RX0uvR5N7Xqv8sAhLsIL92IQkHEIKcm79S7iGDhfkEtw2t9tP5XyeZh5+BHfwBX8Zkfc=</latexit> x1 + x2 <latexit sha1_base64="1coS+Mt9eL/r4jEoYDkZ5NYfoeE=">AAACiHichVHLSsNAFD3Gd3206kZwUywVQSg3KlYFQerGZVutCiqSxGkdTJOQpIVa/AE3LlVcKbgQP8APcOMPuPATxKWCGxfepgFRUW+YzJkz99w5M1d3TOn5RI8tSmtbe0dnV3ekp7evPxobGFzz7IpriIJhm7a7oWueMKUlCr70TbHhuEIr66ZY1/eXGvvrVeF60rZW/ZojtstayZJFaWg+UwV1YmpheieWoBQFEf8J1BAkEEbWjt1iC7uwYaCCMgQs+IxNaPD424QKgsPcNurMuYxksC9wiAhrK5wlOENjdp//JV5thqzF60ZNL1AbfIrJw2VlHEl6oGt6oXu6oSd6/7VWPajR8FLjWW9qhbMTPRpeeftXVebZx96n6k/PPoqYDbxK9u4ETOMWRlNfPTh5WZnPJ+tjdEnP7P+CHumOb2BVX42rnMif/+FHZy/8Ytwg9Xs7foK1yZQ6k5rKTScWM2GrujCCUYxzP9JYxDKyKHB9iWOc4kyJKKSklblmqtISaobwJZTMB48DkKc=</latexit> 1 + 3 = 4 <latexit sha1_base64="Jl1LVcl+mtrFJMn2AKjNmRINEM4=">AAACiHichVHLSsNAFD3GV62vqhvBjVgUQSi39VEVhFI3LtVaW1CRJE7rYJqEJC3U4g+4caniSsGF+AF+gBt/wIWfIC4ruHHhbRoQFfWGyZw5c8+dM3M125CuR/TUorS2tXd0hrrC3T29ff2RgcFN1yo7usjqlmE5eU11hSFNkfWkZ4i87Qi1pBkipx0sN/ZzFeG40jI3vKotdkpq0ZQFqaseU9nE1OxScjcSpRj5MfoTxAMQRRCrVuQO29iDBR1llCBgwmNsQIXL3xbiINjM7aDGnMNI+vsCRwiztsxZgjNUZg/4X+TVVsCavG7UdH21zqcYPBxWjmKcHumG6vRAt/RM77/Wqvk1Gl6qPGtNrbB3+4+HM2//qko8e9j/VP3p2UMB875Xyd5tn2ncQm/qK4en9czi+nhtgq7ohf1f0hPd8w3Myqt+vSbWL/7wo7EXfjFuUPx7O36CzUQsPhebXpuJptJBq0IYwRgmuR9JpLCCVWS5vsQJznCuhBVSkspCM1VpCTRD+BJK+gOby5Ct</latexit> 2 + 5 = 7 <latexit sha1_base64="NRtmyWkgZ5nlm1ETxVkZxPMO8Rc=">AAACiXichVHNLgNRGD0df6V+io3ERjREImnuUJREIrqxpNWSIM3MuGV0/jJz24TGC1jZCVYkFuIBPICNF7DwCGJZiY2Fb6aTCKK+yZ177rnf+e6591MdQ/cEY88RqaW1rb0j2tkV6+7p7Yv3DxQ8u+JqPK/Zhu1uqorHDd3ieaELg286LldM1eAbajnj729UuevptrUuDh2+Yyp7ll7SNUUQVZiZnF+UU8V4giVZECO/gRyCBMJYteP32MYubGiowASHBUHYgAKPvi3IYHCI20GNOJeQHuxzHKOLtBXK4pShEFum/x6ttkLWorVf0wvUGp1i0HBJOYIx9sRuWZ09sjv2wj7+rFULavheDmlWG1ruFPtOhnLv/6pMmgX2v1RNPQuUkA686uTdCRj/FlpDXz06q+cWsmO1cXbNXsn/FXtmD3QDq/qm3azx7GUTPyp5oRejBsk/2/EbFKaS8mxyei2VWFoOWxXFMEYxQf2YwxJWsIo81T/AKc5xIcUkWUpLC41UKRJqBvEtpMwnMx2Q7A==</latexit> 5 + 9 = 14 <latexit sha1_base64="wFMJb33bMt0hW8th8BMIqvUdNU8=">AAACinichVFLSwJRFD5OLzNLq03QRhIjCOSMRk8CqRYtfeQDTGRmutrgvJgZBZP+QLtWQa4KWkQ/oB/Qpj/Qwp8QLQ3atOg4DkRJdoY797vfPd+5371HNBTZshE7Hm5kdGx8wjvpm/JPzwSCs3M5S6+bEstKuqKbBVGwmCJrLGvLtsIKhskEVVRYXqzt9/bzDWZasq4d2U2DlVShqskVWRJsovLxVR53+Xg5GMYoOhEaBLwLwuBGUg8+wjGcgA4S1EEFBhrYhBUQwKKvCDwgGMSVoEWcSUh29hmcg4+0dcpilCEQW6N/lVZFl9Vo3atpOWqJTlFomKQMQQRf8B67+IwP+Iqff9ZqOTV6Xpo0i30tM8qBi4XMx78qlWYbTr9VQz3bUIFNx6tM3g2H6d1C6usbZ1fdzHY60lrGW3wj/zfYwSe6gdZ4l+5SLN0e4kckL/Ri1CD+dzsGQS4W5dej8dRaOLHntsoLi7AEK9SPDUjAISQh69S/hGtoc34uxm1xO/1UzuNq5uFHcAdfqESRGw==</latexit> 3 + 10 = 13 <latexit sha1_base64="FoI7hqDrtQtFjlgLPgriDwtw1iU=">AAACiXichVHLSsNAFD2Nr1pfVTeCm2JRBKFMfNQiCKIbl33YVlApSRx1NE1CMi1o8QdcuRN1peBC/AA/wI0/4MJPEJcV3LjwNg2IFvWGyZw5c8+dM3N1xxSeZOw5pLS1d3R2hbsjPb19/QPRwaGCZ1dcg+cN27TddV3zuCksnpdCmnzdcblW1k1e1A9WGvvFKnc9YVtr8tDhW2Vt1xI7wtAkUYW5qeSiqpaicZZgfsRagRqAOIJI29F7bGIbNgxUUAaHBUnYhAaPvg2oYHCI20KNOJeQ8Pc5jhEhbYWyOGVoxB7Qf5dWGwFr0bpR0/PVBp1i0nBJGcM4e2K3rM4e2R17YR+/1qr5NRpeDmnWm1rulAZORnLv/6rKNEvsfan+9Cyxg5TvVZB3x2catzCa+urRWT23kB2vTbBr9kr+r9gze6AbWNU34ybDs5d/+NHJC70YNUj92Y5WUJhOqMnETGY2vrQctCqMUYxhkvoxjyWsIo081d/HKc5xofQoqpJSFpqpSijQDONbKCufJlSQ5g==</latexit> 5 + 6 = 11 <latexit sha1_base64="n3Yy/9dJfdSB5yYDzUvImVPJv/g=">AAACiHichVHLSsNAFD3Gd3206kZwIxZFEMqNii8QRTcu1VotVClJnNahaRKStKDFH3DjUsWVggvxA/wAN/6Ai36CuFRw48LbNCAq1hsmc+bMPXfOzNUdU3o+UbVJaW5pbWvv6Ix0dff0RmN9/dueXXINkTJs03bTuuYJU1oi5UvfFGnHFVpRN8WOXlit7e+UhetJ29ryDx2xV9TylsxJQ/OZSqkT6uJSNhanBAUx/BuoIYgjjHU7do9d7MOGgRKKELDgMzahweMvAxUEh7k9VJhzGclgX+AYEdaWOEtwhsZsgf95XmVC1uJ1raYXqA0+xeThsnIYo/REt/RKj3RHz/TxZ61KUKPm5ZBnva4VTjZ6Mph8/1dV5NnHwZeqoWcfOcwFXiV7dwKmdgujri8fnb0mFzZHK2N0TS/s/4qq9MA3sMpvxs2G2Lxs4EdnL/xi3CD1Zzt+g+3JhDqTmNqYji+vhK3qwBBGMM79mMUy1rCOFNeXOMU5LpSIQsqsMl9PVZpCzQC+hbLyCaIfkLA=</latexit> 1 + 1 =? <latexit sha1_base64="hRdVP+e2y4bjP1zOby3ZyPytH4w=">AAACi3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKRRhqa+gaatraxwsoG+gZgIECJsMQylBmgIKAfIHtDDEMKQz5DMkMpQy5DKkMeQwlQHYOQyJDMRBGMxgyGDAUAMViGaqBYkVAViZYPpWhloELqLcUqCoVqCIRKJoNJNOBvGioaB6QDzKzGKw7GWhLDhAXAXUqMKgaXDVYafDZ4ITBaoOXBn9wmlUNNgPklkognQTRm1oQz98lEfydoK5cIF3CkIHQhdfNJQxpDBZgt2YC3V4AFgH5Ihmiv6xq+udgqyDVajWDRQavge5faHDT4DDQB3llX5KXBqYGzcbjniSgW4AhBowgQ/TowGSEGekZmukZB5ooOzhBo4qDQZpBiUEDGB/mDA4MHgwBDKHgeJjEMJthDhMvkzGTFZMNRCkTI1SPMAMKYHIFACGukUw=</latexit> 1 + ( 1) =? <latexit sha1_base64="iSckyhI7u8UoVI74y32YMTkgc7c=">AAACj3ichVHLSsNAFD3Gd3201Y3gRiwVQSiTVmwrqEU3umutVUFFkjitoWkSkmlBiz/gB+jChQ9wIX6AH+DGH3DhJ4jLCm5ceJsGREW9YTJnztxz58xc1TZ0VzD21Ca1d3R2dff0Bvr6BwaDofDQumtVHY0XNMuwnE1Vcbmhm7wgdGHwTdvhSkU1+IZaXmrub9S44+qWuSYObL5TUUqmXtQ1RRC1LcdT6fhUOpVIzi3shiIsxrwY+wlkH0TgR9YK3WEbe7CgoYoKOEwIwgYUuPRtQQaDTdwO6sQ5hHRvn+MIAdJWKYtThkJsmf4lWm35rEnrZk3XU2t0ikHDIeUYouyR3bAGe2C37Jm9/1qr7tVoejmgWW1pub0bPB7Jv/2rqtAssP+p+tOzQBEpz6tO3m2Pad5Ca+lrh6eN/OxqtD7BrtgL+b9kT+yebmDWXrXrHF89+8OPSl7oxahB8vd2/ATr8Zg8E0vkpiOZRb9VPRjFOCapH0lksIwsClTfxgnOcSGFpaQ0L2VaqVKbrxnGl5BWPgDiqZJ1</latexit> 12892 + 9837 =? <latexit sha1_base64="3cCCvxejuSwhH3gAGZntIRJA/Ck=">AAAChHichVHNLgNRFP46/uunxUZi02iIhTRnWtWOhTRsLFsUCSIz42LS6cxkZtqExguwJRZWJBbiATyAjRew8AhiSWJj4cy0IhY4N/fec79zvnO/e4/mmIbnEz1FpLb2js6u7p5ob1//QCw+OLTq2TVXF2XdNm13XVM9YRqWKPuGb4p1xxVqVTPFmlZZCOJrdeF6hm2t+AeO2Kqqe5axa+iqz1ApvR1PUkrJZ7OZfIJSREpaybGjKIqckxMyI4El0bKiHb/DJnZgQ0cNVQhY8Nk3ocLjsQEZBIexLTQYc9kzwrjAEaLMrXGW4AyV0Qqve3zaaKEWn4OaXsjW+RaTp8vMBMbpkW7olR7olp7p49dajbBGoOWAd63JFc527Hhk+f1fVpV3H/vfrD81+9hFPtRqsHYnRIJX6E1+/fD8dXl2abwxQVf0wvov6Ynu+QVW/U2/Lomliz/0aKyFf4wb9NWFxO/Oajolz6QypelkYb7Vqm6MYgyT3I8cClhEEWWuL3CCU5xJndKUlJGyzVQp0uIM44dJc58RFJAW</latexit> 2 <latexit sha1_base64="dtxzgykiOKrKKToFg2z7LT9XiS0=">AAAChHichVHNLgNRFP6Mv/ovNhKbRkMspDnTqnYsRNhYaikSRGbGbU1MZyYz0ybVeAG2xMKKxEI8gAew8QIWfQSxJLGxcGZaEQucm3vvud8537nfvUdzTMPziRptUntHZ1d3pKe3r39gcCg6PLLh2RVXFwXdNm13S1M9YRqWKPiGb4otxxVqWTPFpna4HMQ3q8L1DNta92uO2C2rJcsoGrrqM5SjvWicEko2nU5lY5QgUpJKhh1FUeSMHJMZCSyOlq3a0XvsYB82dFRQhoAFn30TKjwe25BBcBjbRZ0xlz0jjAsco5e5Fc4SnKEyeshriU/bLdTic1DTC9k632LydJkZwyQ90S290iPd0TN9/FqrHtYItNR415pc4ewNnYytvf/LKvPu4+Cb9admH0VkQ60Ga3dCJHiF3uRXjy5e1+bzk/UpuqYX1n9FDXrgF1jVN/0mJ/KXf+jRWAv/GDfoqwux352NZEKeS6Rys/HFpVarIhjHBKa5HxksYgWrKHB9gVOc4VzqkmaklJRupkptLc4ofpi08AkM1JAU</latexit> 0 <latexit sha1_base64="P9nwiNSxLZKIKI1oebJys0uGCoU=">AAACiHichVHLLgRBFD3ae7wGG4nNxIRYTW63R09bydhYjscgQaS7FSp6ujvdPZMw8QM2logViYX4AB9g4wcsfIJYkthYuN0zIha4laq6de49t07VtXxHhhHRU5PS3NLa1t7Rmerq7untS/cPrIReJbBFyfYcL1izzFA40hWlSEaOWPMDYZYtR6xa+3NxfLUqglB67nJ04IvNsrnryh1pmxFDJU3TNWMrnaWckZ+amshnKEdkaIbOjmEYqq5mVEZiy6JhRS99hw1sw4ONCsoQcBGx78BEyGMdKgg+Y5uoMRawJ5O4wBFSzK1wluAMk9F9Xnf5tN5AXT7HNcOEbfMtDs+AmRmM0iPd0Cs90C0908evtWpJjVjLAe9WnSv8rb7joaX3f1ll3iPsfbP+1BxhB/lEq2TtfoLEr7Dr/Orh6evSzOJobYyu6IX1X9IT3fML3Oqbfb0gFi/+0GOxFv4xbtBXFzK/OytaTp3OTSxMZmcLjVZ1YBgjGOd+6JjFPIoocX2JE5zhXEkppOiKUU9VmhqcQfwwpfAJa7yREg==</latexit> 22729 train test (the case where a clear answer and inductive logic exist) Try to teach ML “arithmetic addition” only by examples.
23. / 35 12 A toy example: approximate adders RF says 1 + 1 = 5.15 and 1 - 1 = 5.15 and 12892 + 9837 = 12.75 MLP better? But anyway it’s totally wrong. 2-10-5-1 Feed Forward NN (MLP, Multi-Layer Perceptron) adder <latexit sha1_base64="tiacEhQgmTkomNeV5LHmrY6hmbk=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTXxhXRDcuecgjQULaOmJDaZu2EJH4AyZuZeFKExfGD/AD3PgDLvgE4xITNy68lCZGiXib6Zw5c8+dM3NlU1Nth7GOTxgZHRuf8E8GpqZnZoOhufmcbdQthWcVQzOsgizZXFN1nnVUR+MF0+JSTdZ4Xq7u9fbzDW7ZqqEfOE2Tl2pSRVePVUVyiMqclsVyKMJizI3wIBA9EIEXSSP0iEMcwYCCOmrg0OEQ1iDBpq8IEQwmcSW0iLMIqe4+xzkCpK1TFqcMidgq/Su0KnqsTuteTdtVK3SKRsMiZRhR9sLuWZc9swf2yj7/rNVya/S8NGmW+1puloMXi5mPf1U1mh2cfKuGenZwjG3Xq0reTZfp3ULp6xtn7W5mJx1trbBb9kb+b1iHPdEN9Ma7cpfi6eshfmTyQi9GDRJ/t2MQ5NZi4lZsPbURSex6rfJjCctYpX7EkcA+kshS/QoucYW24BdiwqYQ76cKPk+zgB8hJL4AVGCQmw==</latexit> x1 <latexit sha1_base64="4Sn0JXQ8Nli9zYaRMiYfd4a9JHg=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqclmPlQIhFmBPBYRB1QQhuJPXAI45wDB0yGqiDQ4NNWIUIi74iomAwiCuhTZxJSHH2Oc7hI22DsjhliMTW6F+lVdFlNVr3a1qOWqZTVBomKYMIsxd2z3rsmT2wV/b5Z622U6PvpUWzNNByo+y/WM58/Kuq02zj5Fs10rONCnYcrwp5Nxymfwt5oG+edXqZ3XS4vcZu2Rv5v2Fd9kQ30Jrv8l2Kp69H+JHIC70YNSj6ux3DIBeLRLcjG6nNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFVoCQnA==</latexit> x2 <latexit sha1_base64="SKy45aVsIvNNIc9wr1Q6tiMY1uI=">AAACinichVHLSsNAFL2Nr1qrrboR3BRLRRDKpIrPTVEXLvuwD6glJHFah+ZFkpbW4g+4cyXYlYIL8QP8ADf+gIt+gris4MaFN2lAtFhvmMyZM/fcOTNXMhRm2YR0fdzI6Nj4hH8yMBWcngmFZ+fyll43ZZqTdUU3i5JoUYVpNGczW6FFw6SiKim0INX2nf1Cg5oW07Uju2XQsipWNVZhsmgjVWgK/GpTSAjhKIkTNyKDgPdAFLxI6eFHOIYT0EGGOqhAQQMbsQIiWPiVgAcCBnJlaCNnImLuPoVzCKC2jlkUM0Rka/iv4qrksRqunZqWq5bxFAWHicoIxMgLuSc98kweyCv5/LNW263heGnhLPW11BBCFwvZj39VKs42nH6rhnq2oQJbrleG3g2XcW4h9/WNs6tedicTay+TW/KG/m9IlzzhDbTGu3yXppnOED8SesEXwwbxv9sxCPKJOL8RX0uvR5N7Xqv8sAhLsIL92IQkHEIKcm79S7iGDhfkEtw2t9tP5XyeZh5+BHfwBX8Zkfc=</latexit> x1 + x2 <latexit sha1_base64="1coS+Mt9eL/r4jEoYDkZ5NYfoeE=">AAACiHichVHLSsNAFD3Gd3206kZwUywVQSg3KlYFQerGZVutCiqSxGkdTJOQpIVa/AE3LlVcKbgQP8APcOMPuPATxKWCGxfepgFRUW+YzJkz99w5M1d3TOn5RI8tSmtbe0dnV3ekp7evPxobGFzz7IpriIJhm7a7oWueMKUlCr70TbHhuEIr66ZY1/eXGvvrVeF60rZW/ZojtstayZJFaWg+UwV1YmpheieWoBQFEf8J1BAkEEbWjt1iC7uwYaCCMgQs+IxNaPD424QKgsPcNurMuYxksC9wiAhrK5wlOENjdp//JV5thqzF60ZNL1AbfIrJw2VlHEl6oGt6oXu6oSd6/7VWPajR8FLjWW9qhbMTPRpeeftXVebZx96n6k/PPoqYDbxK9u4ETOMWRlNfPTh5WZnPJ+tjdEnP7P+CHumOb2BVX42rnMif/+FHZy/8Ytwg9Xs7foK1yZQ6k5rKTScWM2GrujCCUYxzP9JYxDKyKHB9iWOc4kyJKKSklblmqtISaobwJZTMB48DkKc=</latexit> 1 + 3 = 4 <latexit sha1_base64="Jl1LVcl+mtrFJMn2AKjNmRINEM4=">AAACiHichVHLSsNAFD3GV62vqhvBjVgUQSi39VEVhFI3LtVaW1CRJE7rYJqEJC3U4g+4caniSsGF+AF+gBt/wIWfIC4ruHHhbRoQFfWGyZw5c8+dM3M125CuR/TUorS2tXd0hrrC3T29ff2RgcFN1yo7usjqlmE5eU11hSFNkfWkZ4i87Qi1pBkipx0sN/ZzFeG40jI3vKotdkpq0ZQFqaseU9nE1OxScjcSpRj5MfoTxAMQRRCrVuQO29iDBR1llCBgwmNsQIXL3xbiINjM7aDGnMNI+vsCRwiztsxZgjNUZg/4X+TVVsCavG7UdH21zqcYPBxWjmKcHumG6vRAt/RM77/Wqvk1Gl6qPGtNrbB3+4+HM2//qko8e9j/VP3p2UMB875Xyd5tn2ncQm/qK4en9czi+nhtgq7ohf1f0hPd8w3Myqt+vSbWL/7wo7EXfjFuUPx7O36CzUQsPhebXpuJptJBq0IYwRgmuR9JpLCCVWS5vsQJznCuhBVSkspCM1VpCTRD+BJK+gOby5Ct</latexit> 2 + 5 = 7 <latexit sha1_base64="NRtmyWkgZ5nlm1ETxVkZxPMO8Rc=">AAACiXichVHNLgNRGD0df6V+io3ERjREImnuUJREIrqxpNWSIM3MuGV0/jJz24TGC1jZCVYkFuIBPICNF7DwCGJZiY2Fb6aTCKK+yZ177rnf+e6591MdQ/cEY88RqaW1rb0j2tkV6+7p7Yv3DxQ8u+JqPK/Zhu1uqorHDd3ieaELg286LldM1eAbajnj729UuevptrUuDh2+Yyp7ll7SNUUQVZiZnF+UU8V4giVZECO/gRyCBMJYteP32MYubGiowASHBUHYgAKPvi3IYHCI20GNOJeQHuxzHKOLtBXK4pShEFum/x6ttkLWorVf0wvUGp1i0HBJOYIx9sRuWZ09sjv2wj7+rFULavheDmlWG1ruFPtOhnLv/6pMmgX2v1RNPQuUkA686uTdCRj/FlpDXz06q+cWsmO1cXbNXsn/FXtmD3QDq/qm3azx7GUTPyp5oRejBsk/2/EbFKaS8mxyei2VWFoOWxXFMEYxQf2YwxJWsIo81T/AKc5xIcUkWUpLC41UKRJqBvEtpMwnMx2Q7A==</latexit> 5 + 9 = 14 <latexit sha1_base64="wFMJb33bMt0hW8th8BMIqvUdNU8=">AAACinichVFLSwJRFD5OLzNLq03QRhIjCOSMRk8CqRYtfeQDTGRmutrgvJgZBZP+QLtWQa4KWkQ/oB/Qpj/Qwp8QLQ3atOg4DkRJdoY797vfPd+5371HNBTZshE7Hm5kdGx8wjvpm/JPzwSCs3M5S6+bEstKuqKbBVGwmCJrLGvLtsIKhskEVVRYXqzt9/bzDWZasq4d2U2DlVShqskVWRJsovLxVR53+Xg5GMYoOhEaBLwLwuBGUg8+wjGcgA4S1EEFBhrYhBUQwKKvCDwgGMSVoEWcSUh29hmcg4+0dcpilCEQW6N/lVZFl9Vo3atpOWqJTlFomKQMQQRf8B67+IwP+Iqff9ZqOTV6Xpo0i30tM8qBi4XMx78qlWYbTr9VQz3bUIFNx6tM3g2H6d1C6usbZ1fdzHY60lrGW3wj/zfYwSe6gdZ4l+5SLN0e4kckL/Ri1CD+dzsGQS4W5dej8dRaOLHntsoLi7AEK9SPDUjAISQh69S/hGtoc34uxm1xO/1UzuNq5uFHcAdfqESRGw==</latexit> 3 + 10 = 13 <latexit sha1_base64="FoI7hqDrtQtFjlgLPgriDwtw1iU=">AAACiXichVHLSsNAFD2Nr1pfVTeCm2JRBKFMfNQiCKIbl33YVlApSRx1NE1CMi1o8QdcuRN1peBC/AA/wI0/4MJPEJcV3LjwNg2IFvWGyZw5c8+dM3N1xxSeZOw5pLS1d3R2hbsjPb19/QPRwaGCZ1dcg+cN27TddV3zuCksnpdCmnzdcblW1k1e1A9WGvvFKnc9YVtr8tDhW2Vt1xI7wtAkUYW5qeSiqpaicZZgfsRagRqAOIJI29F7bGIbNgxUUAaHBUnYhAaPvg2oYHCI20KNOJeQ8Pc5jhEhbYWyOGVoxB7Qf5dWGwFr0bpR0/PVBp1i0nBJGcM4e2K3rM4e2R17YR+/1qr5NRpeDmnWm1rulAZORnLv/6rKNEvsfan+9Cyxg5TvVZB3x2catzCa+urRWT23kB2vTbBr9kr+r9gze6AbWNU34ybDs5d/+NHJC70YNUj92Y5WUJhOqMnETGY2vrQctCqMUYxhkvoxjyWsIo081d/HKc5xofQoqpJSFpqpSijQDONbKCufJlSQ5g==</latexit> 5 + 6 = 11 <latexit sha1_base64="n3Yy/9dJfdSB5yYDzUvImVPJv/g=">AAACiHichVHLSsNAFD3Gd3206kZwIxZFEMqNii8QRTcu1VotVClJnNahaRKStKDFH3DjUsWVggvxA/wAN/6Ai36CuFRw48LbNCAq1hsmc+bMPXfOzNUdU3o+UbVJaW5pbWvv6Ix0dff0RmN9/dueXXINkTJs03bTuuYJU1oi5UvfFGnHFVpRN8WOXlit7e+UhetJ29ryDx2xV9TylsxJQ/OZSqkT6uJSNhanBAUx/BuoIYgjjHU7do9d7MOGgRKKELDgMzahweMvAxUEh7k9VJhzGclgX+AYEdaWOEtwhsZsgf95XmVC1uJ1raYXqA0+xeThsnIYo/REt/RKj3RHz/TxZ61KUKPm5ZBnva4VTjZ6Mph8/1dV5NnHwZeqoWcfOcwFXiV7dwKmdgujri8fnb0mFzZHK2N0TS/s/4qq9MA3sMpvxs2G2Lxs4EdnL/xi3CD1Zzt+g+3JhDqTmNqYji+vhK3qwBBGMM79mMUy1rCOFNeXOMU5LpSIQsqsMl9PVZpCzQC+hbLyCaIfkLA=</latexit> 1 + 1 =? <latexit sha1_base64="hRdVP+e2y4bjP1zOby3ZyPytH4w=">AAACi3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKRRhqa+gaatraxwsoG+gZgIECJsMQylBmgIKAfIHtDDEMKQz5DMkMpQy5DKkMeQwlQHYOQyJDMRBGMxgyGDAUAMViGaqBYkVAViZYPpWhloELqLcUqCoVqCIRKJoNJNOBvGioaB6QDzKzGKw7GWhLDhAXAXUqMKgaXDVYafDZ4ITBaoOXBn9wmlUNNgPklkognQTRm1oQz98lEfydoK5cIF3CkIHQhdfNJQxpDBZgt2YC3V4AFgH5Ihmiv6xq+udgqyDVajWDRQavge5faHDT4DDQB3llX5KXBqYGzcbjniSgW4AhBowgQ/TowGSEGekZmukZB5ooOzhBo4qDQZpBiUEDGB/mDA4MHgwBDKHgeJjEMJthDhMvkzGTFZMNRCkTI1SPMAMKYHIFACGukUw=</latexit> 1 + ( 1) =? <latexit sha1_base64="iSckyhI7u8UoVI74y32YMTkgc7c=">AAACj3ichVHLSsNAFD3Gd3201Y3gRiwVQSiTVmwrqEU3umutVUFFkjitoWkSkmlBiz/gB+jChQ9wIX6AH+DGH3DhJ4jLCm5ceJsGREW9YTJnztxz58xc1TZ0VzD21Ca1d3R2dff0Bvr6BwaDofDQumtVHY0XNMuwnE1Vcbmhm7wgdGHwTdvhSkU1+IZaXmrub9S44+qWuSYObL5TUUqmXtQ1RRC1LcdT6fhUOpVIzi3shiIsxrwY+wlkH0TgR9YK3WEbe7CgoYoKOEwIwgYUuPRtQQaDTdwO6sQ5hHRvn+MIAdJWKYtThkJsmf4lWm35rEnrZk3XU2t0ikHDIeUYouyR3bAGe2C37Jm9/1qr7tVoejmgWW1pub0bPB7Jv/2rqtAssP+p+tOzQBEpz6tO3m2Pad5Ca+lrh6eN/OxqtD7BrtgL+b9kT+yebmDWXrXrHF89+8OPSl7oxahB8vd2/ATr8Zg8E0vkpiOZRb9VPRjFOCapH0lksIwsClTfxgnOcSGFpaQ0L2VaqVKbrxnGl5BWPgDiqZJ1</latexit> 12892 + 9837 =? <latexit sha1_base64="3cCCvxejuSwhH3gAGZntIRJA/Ck=">AAAChHichVHNLgNRFP46/uunxUZi02iIhTRnWtWOhTRsLFsUCSIz42LS6cxkZtqExguwJRZWJBbiATyAjRew8AhiSWJj4cy0IhY4N/fec79zvnO/e4/mmIbnEz1FpLb2js6u7p5ob1//QCw+OLTq2TVXF2XdNm13XVM9YRqWKPuGb4p1xxVqVTPFmlZZCOJrdeF6hm2t+AeO2Kqqe5axa+iqz1ApvR1PUkrJZ7OZfIJSREpaybGjKIqckxMyI4El0bKiHb/DJnZgQ0cNVQhY8Nk3ocLjsQEZBIexLTQYc9kzwrjAEaLMrXGW4AyV0Qqve3zaaKEWn4OaXsjW+RaTp8vMBMbpkW7olR7olp7p49dajbBGoOWAd63JFc527Hhk+f1fVpV3H/vfrD81+9hFPtRqsHYnRIJX6E1+/fD8dXl2abwxQVf0wvov6Ynu+QVW/U2/Lomliz/0aKyFf4wb9NWFxO/Oajolz6QypelkYb7Vqm6MYgyT3I8cClhEEWWuL3CCU5xJndKUlJGyzVQp0uIM44dJc58RFJAW</latexit> 2 <latexit sha1_base64="dtxzgykiOKrKKToFg2z7LT9XiS0=">AAAChHichVHNLgNRFP6Mv/ovNhKbRkMspDnTqnYsRNhYaikSRGbGbU1MZyYz0ybVeAG2xMKKxEI8gAew8QIWfQSxJLGxcGZaEQucm3vvud8537nfvUdzTMPziRptUntHZ1d3pKe3r39gcCg6PLLh2RVXFwXdNm13S1M9YRqWKPiGb4otxxVqWTPFpna4HMQ3q8L1DNta92uO2C2rJcsoGrrqM5SjvWicEko2nU5lY5QgUpJKhh1FUeSMHJMZCSyOlq3a0XvsYB82dFRQhoAFn30TKjwe25BBcBjbRZ0xlz0jjAsco5e5Fc4SnKEyeshriU/bLdTic1DTC9k632LydJkZwyQ90S290iPd0TN9/FqrHtYItNR415pc4ewNnYytvf/LKvPu4+Cb9admH0VkQ60Ga3dCJHiF3uRXjy5e1+bzk/UpuqYX1n9FDXrgF1jVN/0mJ/KXf+jRWAv/GDfoqwux352NZEKeS6Rys/HFpVarIhjHBKa5HxksYgWrKHB9gVOc4VzqkmaklJRupkptLc4ofpi08AkM1JAU</latexit> 0 <latexit sha1_base64="P9nwiNSxLZKIKI1oebJys0uGCoU=">AAACiHichVHLLgRBFD3ae7wGG4nNxIRYTW63R09bydhYjscgQaS7FSp6ujvdPZMw8QM2logViYX4AB9g4wcsfIJYkthYuN0zIha4laq6de49t07VtXxHhhHRU5PS3NLa1t7Rmerq7untS/cPrIReJbBFyfYcL1izzFA40hWlSEaOWPMDYZYtR6xa+3NxfLUqglB67nJ04IvNsrnryh1pmxFDJU3TNWMrnaWckZ+amshnKEdkaIbOjmEYqq5mVEZiy6JhRS99hw1sw4ONCsoQcBGx78BEyGMdKgg+Y5uoMRawJ5O4wBFSzK1wluAMk9F9Xnf5tN5AXT7HNcOEbfMtDs+AmRmM0iPd0Cs90C0908evtWpJjVjLAe9WnSv8rb7joaX3f1ll3iPsfbP+1BxhB/lEq2TtfoLEr7Dr/Orh6evSzOJobYyu6IX1X9IT3fML3Oqbfb0gFi/+0GOxFv4xbtBXFzK/OytaTp3OTSxMZmcLjVZ1YBgjGOd+6JjFPIoocX2JE5zhXEkppOiKUU9VmhqcQfwwpfAJa7yREg==</latexit> 22729 train test (the case where a clear answer and inductive logic exist) Try to teach ML “arithmetic addition” only by examples.
24. / 35 13 A toy example: approximate adders Linear regression rocks! 😆 LR says 1 + 1 = 2 and 1 - 1 = 0 and 12892 + 9837 = 22729 adder <latexit sha1_base64="tiacEhQgmTkomNeV5LHmrY6hmbk=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTXxhXRDcuecgjQULaOmJDaZu2EJH4AyZuZeFKExfGD/AD3PgDLvgE4xITNy68lCZGiXib6Zw5c8+dM3NlU1Nth7GOTxgZHRuf8E8GpqZnZoOhufmcbdQthWcVQzOsgizZXFN1nnVUR+MF0+JSTdZ4Xq7u9fbzDW7ZqqEfOE2Tl2pSRVePVUVyiMqclsVyKMJizI3wIBA9EIEXSSP0iEMcwYCCOmrg0OEQ1iDBpq8IEQwmcSW0iLMIqe4+xzkCpK1TFqcMidgq/Su0KnqsTuteTdtVK3SKRsMiZRhR9sLuWZc9swf2yj7/rNVya/S8NGmW+1puloMXi5mPf1U1mh2cfKuGenZwjG3Xq0reTZfp3ULp6xtn7W5mJx1trbBb9kb+b1iHPdEN9Ma7cpfi6eshfmTyQi9GDRJ/t2MQ5NZi4lZsPbURSex6rfJjCctYpX7EkcA+kshS/QoucYW24BdiwqYQ76cKPk+zgB8hJL4AVGCQmw==</latexit> x1 <latexit sha1_base64="4Sn0JXQ8Nli9zYaRMiYfd4a9JHg=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqclmPlQIhFmBPBYRB1QQhuJPXAI45wDB0yGqiDQ4NNWIUIi74iomAwiCuhTZxJSHH2Oc7hI22DsjhliMTW6F+lVdFlNVr3a1qOWqZTVBomKYMIsxd2z3rsmT2wV/b5Z622U6PvpUWzNNByo+y/WM58/Kuq02zj5Fs10rONCnYcrwp5Nxymfwt5oG+edXqZ3XS4vcZu2Rv5v2Fd9kQ30Jrv8l2Kp69H+JHIC70YNSj6ux3DIBeLRLcjG6nNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFVoCQnA==</latexit> x2 <latexit sha1_base64="SKy45aVsIvNNIc9wr1Q6tiMY1uI=">AAACinichVHLSsNAFL2Nr1qrrboR3BRLRRDKpIrPTVEXLvuwD6glJHFah+ZFkpbW4g+4cyXYlYIL8QP8ADf+gIt+gris4MaFN2lAtFhvmMyZM/fcOTNXMhRm2YR0fdzI6Nj4hH8yMBWcngmFZ+fyll43ZZqTdUU3i5JoUYVpNGczW6FFw6SiKim0INX2nf1Cg5oW07Uju2XQsipWNVZhsmgjVWgK/GpTSAjhKIkTNyKDgPdAFLxI6eFHOIYT0EGGOqhAQQMbsQIiWPiVgAcCBnJlaCNnImLuPoVzCKC2jlkUM0Rka/iv4qrksRqunZqWq5bxFAWHicoIxMgLuSc98kweyCv5/LNW263heGnhLPW11BBCFwvZj39VKs42nH6rhnq2oQJbrleG3g2XcW4h9/WNs6tedicTay+TW/KG/m9IlzzhDbTGu3yXppnOED8SesEXwwbxv9sxCPKJOL8RX0uvR5N7Xqv8sAhLsIL92IQkHEIKcm79S7iGDhfkEtw2t9tP5XyeZh5+BHfwBX8Zkfc=</latexit> x1 + x2 <latexit sha1_base64="1coS+Mt9eL/r4jEoYDkZ5NYfoeE=">AAACiHichVHLSsNAFD3Gd3206kZwUywVQSg3KlYFQerGZVutCiqSxGkdTJOQpIVa/AE3LlVcKbgQP8APcOMPuPATxKWCGxfepgFRUW+YzJkz99w5M1d3TOn5RI8tSmtbe0dnV3ekp7evPxobGFzz7IpriIJhm7a7oWueMKUlCr70TbHhuEIr66ZY1/eXGvvrVeF60rZW/ZojtstayZJFaWg+UwV1YmpheieWoBQFEf8J1BAkEEbWjt1iC7uwYaCCMgQs+IxNaPD424QKgsPcNurMuYxksC9wiAhrK5wlOENjdp//JV5thqzF60ZNL1AbfIrJw2VlHEl6oGt6oXu6oSd6/7VWPajR8FLjWW9qhbMTPRpeeftXVebZx96n6k/PPoqYDbxK9u4ETOMWRlNfPTh5WZnPJ+tjdEnP7P+CHumOb2BVX42rnMif/+FHZy/8Ytwg9Xs7foK1yZQ6k5rKTScWM2GrujCCUYxzP9JYxDKyKHB9iWOc4kyJKKSklblmqtISaobwJZTMB48DkKc=</latexit> 1 + 3 = 4 <latexit sha1_base64="Jl1LVcl+mtrFJMn2AKjNmRINEM4=">AAACiHichVHLSsNAFD3GV62vqhvBjVgUQSi39VEVhFI3LtVaW1CRJE7rYJqEJC3U4g+4caniSsGF+AF+gBt/wIWfIC4ruHHhbRoQFfWGyZw5c8+dM3M125CuR/TUorS2tXd0hrrC3T29ff2RgcFN1yo7usjqlmE5eU11hSFNkfWkZ4i87Qi1pBkipx0sN/ZzFeG40jI3vKotdkpq0ZQFqaseU9nE1OxScjcSpRj5MfoTxAMQRRCrVuQO29iDBR1llCBgwmNsQIXL3xbiINjM7aDGnMNI+vsCRwiztsxZgjNUZg/4X+TVVsCavG7UdH21zqcYPBxWjmKcHumG6vRAt/RM77/Wqvk1Gl6qPGtNrbB3+4+HM2//qko8e9j/VP3p2UMB875Xyd5tn2ncQm/qK4en9czi+nhtgq7ohf1f0hPd8w3Myqt+vSbWL/7wo7EXfjFuUPx7O36CzUQsPhebXpuJptJBq0IYwRgmuR9JpLCCVWS5vsQJznCuhBVSkspCM1VpCTRD+BJK+gOby5Ct</latexit> 2 + 5 = 7 <latexit sha1_base64="NRtmyWkgZ5nlm1ETxVkZxPMO8Rc=">AAACiXichVHNLgNRGD0df6V+io3ERjREImnuUJREIrqxpNWSIM3MuGV0/jJz24TGC1jZCVYkFuIBPICNF7DwCGJZiY2Fb6aTCKK+yZ177rnf+e6591MdQ/cEY88RqaW1rb0j2tkV6+7p7Yv3DxQ8u+JqPK/Zhu1uqorHDd3ieaELg286LldM1eAbajnj729UuevptrUuDh2+Yyp7ll7SNUUQVZiZnF+UU8V4giVZECO/gRyCBMJYteP32MYubGiowASHBUHYgAKPvi3IYHCI20GNOJeQHuxzHKOLtBXK4pShEFum/x6ttkLWorVf0wvUGp1i0HBJOYIx9sRuWZ09sjv2wj7+rFULavheDmlWG1ruFPtOhnLv/6pMmgX2v1RNPQuUkA686uTdCRj/FlpDXz06q+cWsmO1cXbNXsn/FXtmD3QDq/qm3azx7GUTPyp5oRejBsk/2/EbFKaS8mxyei2VWFoOWxXFMEYxQf2YwxJWsIo81T/AKc5xIcUkWUpLC41UKRJqBvEtpMwnMx2Q7A==</latexit> 5 + 9 = 14 <latexit sha1_base64="wFMJb33bMt0hW8th8BMIqvUdNU8=">AAACinichVFLSwJRFD5OLzNLq03QRhIjCOSMRk8CqRYtfeQDTGRmutrgvJgZBZP+QLtWQa4KWkQ/oB/Qpj/Qwp8QLQ3atOg4DkRJdoY797vfPd+5371HNBTZshE7Hm5kdGx8wjvpm/JPzwSCs3M5S6+bEstKuqKbBVGwmCJrLGvLtsIKhskEVVRYXqzt9/bzDWZasq4d2U2DlVShqskVWRJsovLxVR53+Xg5GMYoOhEaBLwLwuBGUg8+wjGcgA4S1EEFBhrYhBUQwKKvCDwgGMSVoEWcSUh29hmcg4+0dcpilCEQW6N/lVZFl9Vo3atpOWqJTlFomKQMQQRf8B67+IwP+Iqff9ZqOTV6Xpo0i30tM8qBi4XMx78qlWYbTr9VQz3bUIFNx6tM3g2H6d1C6usbZ1fdzHY60lrGW3wj/zfYwSe6gdZ4l+5SLN0e4kckL/Ri1CD+dzsGQS4W5dej8dRaOLHntsoLi7AEK9SPDUjAISQh69S/hGtoc34uxm1xO/1UzuNq5uFHcAdfqESRGw==</latexit> 3 + 10 = 13 <latexit sha1_base64="FoI7hqDrtQtFjlgLPgriDwtw1iU=">AAACiXichVHLSsNAFD2Nr1pfVTeCm2JRBKFMfNQiCKIbl33YVlApSRx1NE1CMi1o8QdcuRN1peBC/AA/wI0/4MJPEJcV3LjwNg2IFvWGyZw5c8+dM3N1xxSeZOw5pLS1d3R2hbsjPb19/QPRwaGCZ1dcg+cN27TddV3zuCksnpdCmnzdcblW1k1e1A9WGvvFKnc9YVtr8tDhW2Vt1xI7wtAkUYW5qeSiqpaicZZgfsRagRqAOIJI29F7bGIbNgxUUAaHBUnYhAaPvg2oYHCI20KNOJeQ8Pc5jhEhbYWyOGVoxB7Qf5dWGwFr0bpR0/PVBp1i0nBJGcM4e2K3rM4e2R17YR+/1qr5NRpeDmnWm1rulAZORnLv/6rKNEvsfan+9Cyxg5TvVZB3x2catzCa+urRWT23kB2vTbBr9kr+r9gze6AbWNU34ybDs5d/+NHJC70YNUj92Y5WUJhOqMnETGY2vrQctCqMUYxhkvoxjyWsIo081d/HKc5xofQoqpJSFpqpSijQDONbKCufJlSQ5g==</latexit> 5 + 6 = 11 <latexit sha1_base64="n3Yy/9dJfdSB5yYDzUvImVPJv/g=">AAACiHichVHLSsNAFD3Gd3206kZwIxZFEMqNii8QRTcu1VotVClJnNahaRKStKDFH3DjUsWVggvxA/wAN/6Ai36CuFRw48LbNCAq1hsmc+bMPXfOzNUdU3o+UbVJaW5pbWvv6Ix0dff0RmN9/dueXXINkTJs03bTuuYJU1oi5UvfFGnHFVpRN8WOXlit7e+UhetJ29ryDx2xV9TylsxJQ/OZSqkT6uJSNhanBAUx/BuoIYgjjHU7do9d7MOGgRKKELDgMzahweMvAxUEh7k9VJhzGclgX+AYEdaWOEtwhsZsgf95XmVC1uJ1raYXqA0+xeThsnIYo/REt/RKj3RHz/TxZ61KUKPm5ZBnva4VTjZ6Mph8/1dV5NnHwZeqoWcfOcwFXiV7dwKmdgujri8fnb0mFzZHK2N0TS/s/4qq9MA3sMpvxs2G2Lxs4EdnL/xi3CD1Zzt+g+3JhDqTmNqYji+vhK3qwBBGMM79mMUy1rCOFNeXOMU5LpSIQsqsMl9PVZpCzQC+hbLyCaIfkLA=</latexit> 1 + 1 =? <latexit sha1_base64="hRdVP+e2y4bjP1zOby3ZyPytH4w=">AAACi3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKRRhqa+gaatraxwsoG+gZgIECJsMQylBmgIKAfIHtDDEMKQz5DMkMpQy5DKkMeQwlQHYOQyJDMRBGMxgyGDAUAMViGaqBYkVAViZYPpWhloELqLcUqCoVqCIRKJoNJNOBvGioaB6QDzKzGKw7GWhLDhAXAXUqMKgaXDVYafDZ4ITBaoOXBn9wmlUNNgPklkognQTRm1oQz98lEfydoK5cIF3CkIHQhdfNJQxpDBZgt2YC3V4AFgH5Ihmiv6xq+udgqyDVajWDRQavge5faHDT4DDQB3llX5KXBqYGzcbjniSgW4AhBowgQ/TowGSEGekZmukZB5ooOzhBo4qDQZpBiUEDGB/mDA4MHgwBDKHgeJjEMJthDhMvkzGTFZMNRCkTI1SPMAMKYHIFACGukUw=</latexit> 1 + ( 1) =? <latexit sha1_base64="iSckyhI7u8UoVI74y32YMTkgc7c=">AAACj3ichVHLSsNAFD3Gd3201Y3gRiwVQSiTVmwrqEU3umutVUFFkjitoWkSkmlBiz/gB+jChQ9wIX6AH+DGH3DhJ4jLCm5ceJsGREW9YTJnztxz58xc1TZ0VzD21Ca1d3R2dff0Bvr6BwaDofDQumtVHY0XNMuwnE1Vcbmhm7wgdGHwTdvhSkU1+IZaXmrub9S44+qWuSYObL5TUUqmXtQ1RRC1LcdT6fhUOpVIzi3shiIsxrwY+wlkH0TgR9YK3WEbe7CgoYoKOEwIwgYUuPRtQQaDTdwO6sQ5hHRvn+MIAdJWKYtThkJsmf4lWm35rEnrZk3XU2t0ikHDIeUYouyR3bAGe2C37Jm9/1qr7tVoejmgWW1pub0bPB7Jv/2rqtAssP+p+tOzQBEpz6tO3m2Pad5Ca+lrh6eN/OxqtD7BrtgL+b9kT+yebmDWXrXrHF89+8OPSl7oxahB8vd2/ATr8Zg8E0vkpiOZRb9VPRjFOCapH0lksIwsClTfxgnOcSGFpaQ0L2VaqVKbrxnGl5BWPgDiqZJ1</latexit> 12892 + 9837 =? <latexit sha1_base64="3cCCvxejuSwhH3gAGZntIRJA/Ck=">AAAChHichVHNLgNRFP46/uunxUZi02iIhTRnWtWOhTRsLFsUCSIz42LS6cxkZtqExguwJRZWJBbiATyAjRew8AhiSWJj4cy0IhY4N/fec79zvnO/e4/mmIbnEz1FpLb2js6u7p5ob1//QCw+OLTq2TVXF2XdNm13XVM9YRqWKPuGb4p1xxVqVTPFmlZZCOJrdeF6hm2t+AeO2Kqqe5axa+iqz1ApvR1PUkrJZ7OZfIJSREpaybGjKIqckxMyI4El0bKiHb/DJnZgQ0cNVQhY8Nk3ocLjsQEZBIexLTQYc9kzwrjAEaLMrXGW4AyV0Qqve3zaaKEWn4OaXsjW+RaTp8vMBMbpkW7olR7olp7p49dajbBGoOWAd63JFc527Hhk+f1fVpV3H/vfrD81+9hFPtRqsHYnRIJX6E1+/fD8dXl2abwxQVf0wvov6Ynu+QVW/U2/Lomliz/0aKyFf4wb9NWFxO/Oajolz6QypelkYb7Vqm6MYgyT3I8cClhEEWWuL3CCU5xJndKUlJGyzVQp0uIM44dJc58RFJAW</latexit> 2 <latexit sha1_base64="dtxzgykiOKrKKToFg2z7LT9XiS0=">AAAChHichVHNLgNRFP6Mv/ovNhKbRkMspDnTqnYsRNhYaikSRGbGbU1MZyYz0ybVeAG2xMKKxEI8gAew8QIWfQSxJLGxcGZaEQucm3vvud8537nfvUdzTMPziRptUntHZ1d3pKe3r39gcCg6PLLh2RVXFwXdNm13S1M9YRqWKPiGb4otxxVqWTPFpna4HMQ3q8L1DNta92uO2C2rJcsoGrrqM5SjvWicEko2nU5lY5QgUpJKhh1FUeSMHJMZCSyOlq3a0XvsYB82dFRQhoAFn30TKjwe25BBcBjbRZ0xlz0jjAsco5e5Fc4SnKEyeshriU/bLdTic1DTC9k632LydJkZwyQ90S290iPd0TN9/FqrHtYItNR415pc4ewNnYytvf/LKvPu4+Cb9admH0VkQ60Ga3dCJHiF3uRXjy5e1+bzk/UpuqYX1n9FDXrgF1jVN/0mJ/KXf+jRWAv/GDfoqwux352NZEKeS6Rys/HFpVarIhjHBKa5HxksYgWrKHB9gVOc4VzqkmaklJRupkptLc4ofpi08AkM1JAU</latexit> 0 <latexit sha1_base64="P9nwiNSxLZKIKI1oebJys0uGCoU=">AAACiHichVHLLgRBFD3ae7wGG4nNxIRYTW63R09bydhYjscgQaS7FSp6ujvdPZMw8QM2logViYX4AB9g4wcsfIJYkthYuN0zIha4laq6de49t07VtXxHhhHRU5PS3NLa1t7Rmerq7untS/cPrIReJbBFyfYcL1izzFA40hWlSEaOWPMDYZYtR6xa+3NxfLUqglB67nJ04IvNsrnryh1pmxFDJU3TNWMrnaWckZ+amshnKEdkaIbOjmEYqq5mVEZiy6JhRS99hw1sw4ONCsoQcBGx78BEyGMdKgg+Y5uoMRawJ5O4wBFSzK1wluAMk9F9Xnf5tN5AXT7HNcOEbfMtDs+AmRmM0iPd0Cs90C0908evtWpJjVjLAe9WnSv8rb7joaX3f1ll3iPsfbP+1BxhB/lEq2TtfoLEr7Dr/Orh6evSzOJobYyu6IX1X9IT3fML3Oqbfb0gFi/+0GOxFv4xbtBXFzK/OytaTp3OTSxMZmcLjVZ1YBgjGOd+6JjFPIoocX2JE5zhXEkppOiKUU9VmhqcQfwwpfAJa7yREg==</latexit> 22729 train test (Perfect Answers!!)
25. / 35 13 A toy example: approximate adders Linear regression rocks! 😆 LR says 1 + 1 = 2 and 1 - 1 = 0 and 12892 + 9837 = 22729 All are because of “inductive bias” intrinsically encoded in the model. LR with two input variables is just fitting a plane <latexit sha1_base64="iFKbOGXhRe3/O37wsq+JtGPu+ec=">AAAClnichVHLSsNAFD2Nr1pfrW4EN8FSEYQyUVFxIaKILvuwraBSkjitoWkSkrRaiz/gD7gQFwoq4gf4AW78ARf9BHFZwY0Lb9OAqKg3JPfMmXtuzsxVLF1zXMYaAaGjs6u7J9gb6usfGBwKR4azjlmxVZ5RTd20txTZ4bpm8IyruTrfsmwulxWd55TSams/V+W2o5nGpluz+G5ZLhpaQVNll6h8OCKLh3lJnBIVytOU1Xw4yuLMC/EnkHwQhR8JM3yPHezBhIoKyuAw4BLWIcOhZxsSGCzidlEnziakefscxwiRtkJVnCpkYkv0LdJq22cNWrd6Op5apb/o9NqkFBFjT+yWNdkju2PP7P3XXnWvR8tLjbLS1nIrP3Qymn77V1Wm7GL/U/WnZxcFLHheNfJueUzrFGpbXz06baYXU7H6BLtkL+T/gjXYA53AqL6qV0meOvvDj0Je6MZoQNL3cfwE2em4NBefSc5Gl1f8UQUxhnFM0jzmsYwNJJCh/gc4xzVuhFFhSVgT1tulQsDXjOBLCIkPAxKUnQ==</latexit> ax1 + bx2 + c Any three instances are enough to have <latexit sha1_base64="57verNgszw6m8+AhOWc3Hu105bk=">AAACp3ichVFNS9xQFD2mftX6MW03BTfBwaIow42WtgiC6MadOjrjgMrwEp/jw0wSkjdT7eBe/AMuXCm4KAW3unfjH+jCn1C6VHDjojeZgKioN+S+88675+a8XDtwVaSJrlqMN61t7R2db7vedff09mXefyhGfi10ZMHxXT8s2SKSrvJkQSvtylIQSlG1Xblsb83E58t1GUbK95b0TiDXqqLiqQ3lCM1UOTMgJq1R046TM0mreVXZ1CIM/R/mdtkyRziPlTNZylES5lNgpSCLNOb9zBlWsQ4fDmqoQsKDZuxCIOJnBRYIAXNraDAXMlLJucQuulhb4yrJFYLZLc4V3q2krMf7uGeUqB3+istvyEoTg/SHftE1XdJv+kt3z/ZqJD1iLzu82k2tDMp9+58Wb19VVXnV2LxXvehZYwPfE6+KvQcJE9/CaerrPw+uFyfyg43PdEz/2P8RXdEF38Cr3zgnCzJ/+IIfm73wH+MBWY/H8RQUx3LW19z4wpfs1HQ6qk70YwBDPI9vmMIs5lHg/ns4xRnOjWFjzigapWap0ZJqPuJBGOI/dKSbIg==</latexit> a = 1, b = 1, c = 0 ) x1 + x2 MLP has partial “linearity” inside and that’s why MLP is better than RF is. RF is “piecewise constant” and only returns values between sample min and max. to points in 3D adder <latexit sha1_base64="tiacEhQgmTkomNeV5LHmrY6hmbk=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTXxhXRDcuecgjQULaOmJDaZu2EJH4AyZuZeFKExfGD/AD3PgDLvgE4xITNy68lCZGiXib6Zw5c8+dM3NlU1Nth7GOTxgZHRuf8E8GpqZnZoOhufmcbdQthWcVQzOsgizZXFN1nnVUR+MF0+JSTdZ4Xq7u9fbzDW7ZqqEfOE2Tl2pSRVePVUVyiMqclsVyKMJizI3wIBA9EIEXSSP0iEMcwYCCOmrg0OEQ1iDBpq8IEQwmcSW0iLMIqe4+xzkCpK1TFqcMidgq/Su0KnqsTuteTdtVK3SKRsMiZRhR9sLuWZc9swf2yj7/rNVya/S8NGmW+1puloMXi5mPf1U1mh2cfKuGenZwjG3Xq0reTZfp3ULp6xtn7W5mJx1trbBb9kb+b1iHPdEN9Ma7cpfi6eshfmTyQi9GDRJ/t2MQ5NZi4lZsPbURSex6rfJjCctYpX7EkcA+kshS/QoucYW24BdiwqYQ76cKPk+zgB8hJL4AVGCQmw==</latexit> x1 <latexit sha1_base64="4Sn0JXQ8Nli9zYaRMiYfd4a9JHg=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqclmPlQIhFmBPBYRB1QQhuJPXAI45wDB0yGqiDQ4NNWIUIi74iomAwiCuhTZxJSHH2Oc7hI22DsjhliMTW6F+lVdFlNVr3a1qOWqZTVBomKYMIsxd2z3rsmT2wV/b5Z622U6PvpUWzNNByo+y/WM58/Kuq02zj5Fs10rONCnYcrwp5Nxymfwt5oG+edXqZ3XS4vcZu2Rv5v2Fd9kQ30Jrv8l2Kp69H+JHIC70YNSj6ux3DIBeLRLcjG6nNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFVoCQnA==</latexit> x2 <latexit sha1_base64="SKy45aVsIvNNIc9wr1Q6tiMY1uI=">AAACinichVHLSsNAFL2Nr1qrrboR3BRLRRDKpIrPTVEXLvuwD6glJHFah+ZFkpbW4g+4cyXYlYIL8QP8ADf+gIt+gris4MaFN2lAtFhvmMyZM/fcOTNXMhRm2YR0fdzI6Nj4hH8yMBWcngmFZ+fyll43ZZqTdUU3i5JoUYVpNGczW6FFw6SiKim0INX2nf1Cg5oW07Uju2XQsipWNVZhsmgjVWgK/GpTSAjhKIkTNyKDgPdAFLxI6eFHOIYT0EGGOqhAQQMbsQIiWPiVgAcCBnJlaCNnImLuPoVzCKC2jlkUM0Rka/iv4qrksRqunZqWq5bxFAWHicoIxMgLuSc98kweyCv5/LNW263heGnhLPW11BBCFwvZj39VKs42nH6rhnq2oQJbrleG3g2XcW4h9/WNs6tedicTay+TW/KG/m9IlzzhDbTGu3yXppnOED8SesEXwwbxv9sxCPKJOL8RX0uvR5N7Xqv8sAhLsIL92IQkHEIKcm79S7iGDhfkEtw2t9tP5XyeZh5+BHfwBX8Zkfc=</latexit> x1 + x2 <latexit sha1_base64="1coS+Mt9eL/r4jEoYDkZ5NYfoeE=">AAACiHichVHLSsNAFD3Gd3206kZwUywVQSg3KlYFQerGZVutCiqSxGkdTJOQpIVa/AE3LlVcKbgQP8APcOMPuPATxKWCGxfepgFRUW+YzJkz99w5M1d3TOn5RI8tSmtbe0dnV3ekp7evPxobGFzz7IpriIJhm7a7oWueMKUlCr70TbHhuEIr66ZY1/eXGvvrVeF60rZW/ZojtstayZJFaWg+UwV1YmpheieWoBQFEf8J1BAkEEbWjt1iC7uwYaCCMgQs+IxNaPD424QKgsPcNurMuYxksC9wiAhrK5wlOENjdp//JV5thqzF60ZNL1AbfIrJw2VlHEl6oGt6oXu6oSd6/7VWPajR8FLjWW9qhbMTPRpeeftXVebZx96n6k/PPoqYDbxK9u4ETOMWRlNfPTh5WZnPJ+tjdEnP7P+CHumOb2BVX42rnMif/+FHZy/8Ytwg9Xs7foK1yZQ6k5rKTScWM2GrujCCUYxzP9JYxDKyKHB9iWOc4kyJKKSklblmqtISaobwJZTMB48DkKc=</latexit> 1 + 3 = 4 <latexit sha1_base64="Jl1LVcl+mtrFJMn2AKjNmRINEM4=">AAACiHichVHLSsNAFD3GV62vqhvBjVgUQSi39VEVhFI3LtVaW1CRJE7rYJqEJC3U4g+4caniSsGF+AF+gBt/wIWfIC4ruHHhbRoQFfWGyZw5c8+dM3M125CuR/TUorS2tXd0hrrC3T29ff2RgcFN1yo7usjqlmE5eU11hSFNkfWkZ4i87Qi1pBkipx0sN/ZzFeG40jI3vKotdkpq0ZQFqaseU9nE1OxScjcSpRj5MfoTxAMQRRCrVuQO29iDBR1llCBgwmNsQIXL3xbiINjM7aDGnMNI+vsCRwiztsxZgjNUZg/4X+TVVsCavG7UdH21zqcYPBxWjmKcHumG6vRAt/RM77/Wqvk1Gl6qPGtNrbB3+4+HM2//qko8e9j/VP3p2UMB875Xyd5tn2ncQm/qK4en9czi+nhtgq7ohf1f0hPd8w3Myqt+vSbWL/7wo7EXfjFuUPx7O36CzUQsPhebXpuJptJBq0IYwRgmuR9JpLCCVWS5vsQJznCuhBVSkspCM1VpCTRD+BJK+gOby5Ct</latexit> 2 + 5 = 7 <latexit sha1_base64="NRtmyWkgZ5nlm1ETxVkZxPMO8Rc=">AAACiXichVHNLgNRGD0df6V+io3ERjREImnuUJREIrqxpNWSIM3MuGV0/jJz24TGC1jZCVYkFuIBPICNF7DwCGJZiY2Fb6aTCKK+yZ177rnf+e6591MdQ/cEY88RqaW1rb0j2tkV6+7p7Yv3DxQ8u+JqPK/Zhu1uqorHDd3ieaELg286LldM1eAbajnj729UuevptrUuDh2+Yyp7ll7SNUUQVZiZnF+UU8V4giVZECO/gRyCBMJYteP32MYubGiowASHBUHYgAKPvi3IYHCI20GNOJeQHuxzHKOLtBXK4pShEFum/x6ttkLWorVf0wvUGp1i0HBJOYIx9sRuWZ09sjv2wj7+rFULavheDmlWG1ruFPtOhnLv/6pMmgX2v1RNPQuUkA686uTdCRj/FlpDXz06q+cWsmO1cXbNXsn/FXtmD3QDq/qm3azx7GUTPyp5oRejBsk/2/EbFKaS8mxyei2VWFoOWxXFMEYxQf2YwxJWsIo81T/AKc5xIcUkWUpLC41UKRJqBvEtpMwnMx2Q7A==</latexit> 5 + 9 = 14 <latexit sha1_base64="wFMJb33bMt0hW8th8BMIqvUdNU8=">AAACinichVFLSwJRFD5OLzNLq03QRhIjCOSMRk8CqRYtfeQDTGRmutrgvJgZBZP+QLtWQa4KWkQ/oB/Qpj/Qwp8QLQ3atOg4DkRJdoY797vfPd+5371HNBTZshE7Hm5kdGx8wjvpm/JPzwSCs3M5S6+bEstKuqKbBVGwmCJrLGvLtsIKhskEVVRYXqzt9/bzDWZasq4d2U2DlVShqskVWRJsovLxVR53+Xg5GMYoOhEaBLwLwuBGUg8+wjGcgA4S1EEFBhrYhBUQwKKvCDwgGMSVoEWcSUh29hmcg4+0dcpilCEQW6N/lVZFl9Vo3atpOWqJTlFomKQMQQRf8B67+IwP+Iqff9ZqOTV6Xpo0i30tM8qBi4XMx78qlWYbTr9VQz3bUIFNx6tM3g2H6d1C6usbZ1fdzHY60lrGW3wj/zfYwSe6gdZ4l+5SLN0e4kckL/Ri1CD+dzsGQS4W5dej8dRaOLHntsoLi7AEK9SPDUjAISQh69S/hGtoc34uxm1xO/1UzuNq5uFHcAdfqESRGw==</latexit> 3 + 10 = 13 <latexit sha1_base64="FoI7hqDrtQtFjlgLPgriDwtw1iU=">AAACiXichVHLSsNAFD2Nr1pfVTeCm2JRBKFMfNQiCKIbl33YVlApSRx1NE1CMi1o8QdcuRN1peBC/AA/wI0/4MJPEJcV3LjwNg2IFvWGyZw5c8+dM3N1xxSeZOw5pLS1d3R2hbsjPb19/QPRwaGCZ1dcg+cN27TddV3zuCksnpdCmnzdcblW1k1e1A9WGvvFKnc9YVtr8tDhW2Vt1xI7wtAkUYW5qeSiqpaicZZgfsRagRqAOIJI29F7bGIbNgxUUAaHBUnYhAaPvg2oYHCI20KNOJeQ8Pc5jhEhbYWyOGVoxB7Qf5dWGwFr0bpR0/PVBp1i0nBJGcM4e2K3rM4e2R17YR+/1qr5NRpeDmnWm1rulAZORnLv/6rKNEvsfan+9Cyxg5TvVZB3x2catzCa+urRWT23kB2vTbBr9kr+r9gze6AbWNU34ybDs5d/+NHJC70YNUj92Y5WUJhOqMnETGY2vrQctCqMUYxhkvoxjyWsIo081d/HKc5xofQoqpJSFpqpSijQDONbKCufJlSQ5g==</latexit> 5 + 6 = 11 <latexit sha1_base64="n3Yy/9dJfdSB5yYDzUvImVPJv/g=">AAACiHichVHLSsNAFD3Gd3206kZwIxZFEMqNii8QRTcu1VotVClJnNahaRKStKDFH3DjUsWVggvxA/wAN/6Ai36CuFRw48LbNCAq1hsmc+bMPXfOzNUdU3o+UbVJaW5pbWvv6Ix0dff0RmN9/dueXXINkTJs03bTuuYJU1oi5UvfFGnHFVpRN8WOXlit7e+UhetJ29ryDx2xV9TylsxJQ/OZSqkT6uJSNhanBAUx/BuoIYgjjHU7do9d7MOGgRKKELDgMzahweMvAxUEh7k9VJhzGclgX+AYEdaWOEtwhsZsgf95XmVC1uJ1raYXqA0+xeThsnIYo/REt/RKj3RHz/TxZ61KUKPm5ZBnva4VTjZ6Mph8/1dV5NnHwZeqoWcfOcwFXiV7dwKmdgujri8fnb0mFzZHK2N0TS/s/4qq9MA3sMpvxs2G2Lxs4EdnL/xi3CD1Zzt+g+3JhDqTmNqYji+vhK3qwBBGMM79mMUy1rCOFNeXOMU5LpSIQsqsMl9PVZpCzQC+hbLyCaIfkLA=</latexit> 1 + 1 =? <latexit sha1_base64="hRdVP+e2y4bjP1zOby3ZyPytH4w=">AAACi3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKRRhqa+gaatraxwsoG+gZgIECJsMQylBmgIKAfIHtDDEMKQz5DMkMpQy5DKkMeQwlQHYOQyJDMRBGMxgyGDAUAMViGaqBYkVAViZYPpWhloELqLcUqCoVqCIRKJoNJNOBvGioaB6QDzKzGKw7GWhLDhAXAXUqMKgaXDVYafDZ4ITBaoOXBn9wmlUNNgPklkognQTRm1oQz98lEfydoK5cIF3CkIHQhdfNJQxpDBZgt2YC3V4AFgH5Ihmiv6xq+udgqyDVajWDRQavge5faHDT4DDQB3llX5KXBqYGzcbjniSgW4AhBowgQ/TowGSEGekZmukZB5ooOzhBo4qDQZpBiUEDGB/mDA4MHgwBDKHgeJjEMJthDhMvkzGTFZMNRCkTI1SPMAMKYHIFACGukUw=</latexit> 1 + ( 1) =? <latexit sha1_base64="iSckyhI7u8UoVI74y32YMTkgc7c=">AAACj3ichVHLSsNAFD3Gd3201Y3gRiwVQSiTVmwrqEU3umutVUFFkjitoWkSkmlBiz/gB+jChQ9wIX6AH+DGH3DhJ4jLCm5ceJsGREW9YTJnztxz58xc1TZ0VzD21Ca1d3R2dff0Bvr6BwaDofDQumtVHY0XNMuwnE1Vcbmhm7wgdGHwTdvhSkU1+IZaXmrub9S44+qWuSYObL5TUUqmXtQ1RRC1LcdT6fhUOpVIzi3shiIsxrwY+wlkH0TgR9YK3WEbe7CgoYoKOEwIwgYUuPRtQQaDTdwO6sQ5hHRvn+MIAdJWKYtThkJsmf4lWm35rEnrZk3XU2t0ikHDIeUYouyR3bAGe2C37Jm9/1qr7tVoejmgWW1pub0bPB7Jv/2rqtAssP+p+tOzQBEpz6tO3m2Pad5Ca+lrh6eN/OxqtD7BrtgL+b9kT+yebmDWXrXrHF89+8OPSl7oxahB8vd2/ATr8Zg8E0vkpiOZRb9VPRjFOCapH0lksIwsClTfxgnOcSGFpaQ0L2VaqVKbrxnGl5BWPgDiqZJ1</latexit> 12892 + 9837 =? <latexit sha1_base64="3cCCvxejuSwhH3gAGZntIRJA/Ck=">AAAChHichVHNLgNRFP46/uunxUZi02iIhTRnWtWOhTRsLFsUCSIz42LS6cxkZtqExguwJRZWJBbiATyAjRew8AhiSWJj4cy0IhY4N/fec79zvnO/e4/mmIbnEz1FpLb2js6u7p5ob1//QCw+OLTq2TVXF2XdNm13XVM9YRqWKPuGb4p1xxVqVTPFmlZZCOJrdeF6hm2t+AeO2Kqqe5axa+iqz1ApvR1PUkrJZ7OZfIJSREpaybGjKIqckxMyI4El0bKiHb/DJnZgQ0cNVQhY8Nk3ocLjsQEZBIexLTQYc9kzwrjAEaLMrXGW4AyV0Qqve3zaaKEWn4OaXsjW+RaTp8vMBMbpkW7olR7olp7p49dajbBGoOWAd63JFc527Hhk+f1fVpV3H/vfrD81+9hFPtRqsHYnRIJX6E1+/fD8dXl2abwxQVf0wvov6Ynu+QVW/U2/Lomliz/0aKyFf4wb9NWFxO/Oajolz6QypelkYb7Vqm6MYgyT3I8cClhEEWWuL3CCU5xJndKUlJGyzVQp0uIM44dJc58RFJAW</latexit> 2 <latexit sha1_base64="dtxzgykiOKrKKToFg2z7LT9XiS0=">AAAChHichVHNLgNRFP6Mv/ovNhKbRkMspDnTqnYsRNhYaikSRGbGbU1MZyYz0ybVeAG2xMKKxEI8gAew8QIWfQSxJLGxcGZaEQucm3vvud8537nfvUdzTMPziRptUntHZ1d3pKe3r39gcCg6PLLh2RVXFwXdNm13S1M9YRqWKPiGb4otxxVqWTPFpna4HMQ3q8L1DNta92uO2C2rJcsoGrrqM5SjvWicEko2nU5lY5QgUpJKhh1FUeSMHJMZCSyOlq3a0XvsYB82dFRQhoAFn30TKjwe25BBcBjbRZ0xlz0jjAsco5e5Fc4SnKEyeshriU/bLdTic1DTC9k632LydJkZwyQ90S290iPd0TN9/FqrHtYItNR415pc4ewNnYytvf/LKvPu4+Cb9admH0VkQ60Ga3dCJHiF3uRXjy5e1+bzk/UpuqYX1n9FDXrgF1jVN/0mJ/KXf+jRWAv/GDfoqwux352NZEKeS6Rys/HFpVarIhjHBKa5HxksYgWrKHB9gVOc4VzqkmaklJRupkptLc4ofpi08AkM1JAU</latexit> 0 <latexit sha1_base64="P9nwiNSxLZKIKI1oebJys0uGCoU=">AAACiHichVHLLgRBFD3ae7wGG4nNxIRYTW63R09bydhYjscgQaS7FSp6ujvdPZMw8QM2logViYX4AB9g4wcsfIJYkthYuN0zIha4laq6de49t07VtXxHhhHRU5PS3NLa1t7Rmerq7untS/cPrIReJbBFyfYcL1izzFA40hWlSEaOWPMDYZYtR6xa+3NxfLUqglB67nJ04IvNsrnryh1pmxFDJU3TNWMrnaWckZ+amshnKEdkaIbOjmEYqq5mVEZiy6JhRS99hw1sw4ONCsoQcBGx78BEyGMdKgg+Y5uoMRawJ5O4wBFSzK1wluAMk9F9Xnf5tN5AXT7HNcOEbfMtDs+AmRmM0iPd0Cs90C0908evtWpJjVjLAe9WnSv8rb7joaX3f1ll3iPsfbP+1BxhB/lEq2TtfoLEr7Dr/Orh6evSzOJobYyu6IX1X9IT3fML3Oqbfb0gFi/+0GOxFv4xbtBXFzK/OytaTp3OTSxMZmcLjVZ1YBgjGOd+6JjFPIoocX2JE5zhXEkppOiKUU9VmhqcQfwwpfAJa7yREg==</latexit> 22729 train test (Perfect Answers!!)
26. / 35 14 Designing relevant “inductive biases” “Inductive biases” (often unintendedly) defines the interspace of instances, and then also defines the prediction for unseen areas (crucial for out-of-distribution predictions). Neural Networks (ReLU) Random Forest Linear Regression Extra Trees (no bootstrap) Carefully designed inputs (input only confidently relevant info into ML) Conservative models + Strong inductive biases that best fit to the given problem General models having a large number of parameters + Generalizable inductive biases (enables zero-shot/few-shot transfer?) Kitchen-sink inputs (input all potentially relevant info into ML) Use any “physics-informed” conditions to further constrain or regularize the model space sounds a good idea indeed Complex Simple Small Large Input-Output Correlation for Target Data Required To Make ML Work
27. / 35 15 A recent news: OGB Large-Scale Challenge @ KDDCup 2021 Current ML is too data-hungry (and purportedly vulnerable to any data bias) Gaps between technical interests and reality Modern ML can learn any input-output mappings in theory, but more data is needed when the input-output correlation is weak. PCQM4M-LSC predicting DFT-calculated HOMO-LUMO energy gap of molecules given their 2D molecular graphs. (3,803,453 graphs from PubChemQC; cf. 133,885 graphs for QM9) 1st place: 10 GNNs (12-Layer Graphormer) + 8 ExpC*s (5-Layer ExpandingConv) 73 GNNs (11-Layer LiteGEMConv with Self-Supervised Pretraining) 20 GNNs (32-Layer GN with Noisy Nodes) Test MAE 0.1200 (eV) 2nd place: Test MAE 0.1204 (eV) 3rd place: Test MAE 0.1205 (eV) Our reality. Practical “open-end” cases we can only have very limited data relative to the astronomically vast search space. Our technical interests. we’re very excited to explore ML over large data (for pretraining + transfer) with generalizable modular structures: CNNs vs Transformers vs GNNs vs MLPs
28. / 35 16 Today’s talk Gas-phase reactions on sold-phase catalyst surface (Heterogeneous catalysis) Industrial Synthesis (e.g. Haber-Bosch), Automobile Exhaust Gas Purification, Methane Conversion, etc. Reactants (Gas) Catalysts (Solid) Nano-particle surface High Temperature, High Pressure Adsorption Diffusion Dissociation Recombination Desorption God made the bulk; the surface was invented by the devil Devilishly complex too-many-factor process!! —— Wolfgang Pauli Our struggles for better ML practices with underspecified, sparse, biased observational data (i.e. a collection of experimental facts from literature)
29. / 35 17 The base dataset http://www.fhi-berlin.mpg.de/acnew/department/pages/ocmdata.html https://www.nature.com/articles/s41467-019-08325-8#Sec19 Oxidative coupling of methane (OCM) reactions Methane (CH4 ) is partially oxidized to C2 hydrocarbons such as ethane (C2 H6 ) and ethylene (C2 H4 ) in a single step Elemental composition of catalyst (mol%) Process parameters + Preparation Catalytic performance • Zavyalova, U.; Holena, M.; Schlögl, R.; Baerns, ChemCatChem 2011. • Followup: Kondratenko, E. V.; Schlüter, M.; Baerns, M.; Linke, D.; Holena, M. Catal. Sci. Technol. 2015. • Renalysis with Corrections & Outlier Removal Schmack, R.; Friedrich, A.; Kondratenko, E. V.; Polte, J.; Werwatz, A.; Kraehnert, R. Nat Commun 2019. 1866 catalyst records from 421 reports
30. / 35 18 Problem #1: It’s underspecified • Each catalyst was mostly measured at different reaction conditions. • Only a few are measured under multiple conditions 158 compositions > 2 conditions 60 compositions > 3 conditions 26 compositions > 4 conditions La:100.0 x 24 Mg:100.0 x 18 Ca:100.0 x 18 Sm:100.0 x 13 Nd:100.0 x 10 Ba:100.0 x 9 Y:100.0 x 9 Ce:100.0 x 9 Zr:100.0 x 9 Sr:100.0 x 7 Si:100.0 x 7 Gd:100.0 x 7 Na:8.9 Si:83.1 Mn:3.5 W:4.5 x 7 Pr:100.0 x 6 Eu:100.0 x 6 Yb:100.0 x 5 Al:100.0 x 4 Li:10.0 Mg:90.0 x 4 Mg:90.9 La:9.1 x 4 Li:9.1 Mg:90.9 x 4 Tb:100.0 x 4 Li:20.0 Cl:20.0 Zr:60.0 x 4 Na:20.0 Cl:20.0 Zr:60.0 x 4 Cl:20.0 K:20.0 Zr:60.0 x 4 Cl:20.0 Rb:20.0 Zr:60.0 x 4 Cl:20.0 Zr:60.0 Cs:20.0 x 4 • No replicates in the same conditions • But as we see later, reaction conditions are quite influential. Because of this, “no generally valid correlation between a catalyst’s composition, its structure and its OCM performance has been established yet.” Strong limitation of observational data (just passively acquired) Observational study Interventional study
31. / 35 19 Problem #2: It’s sparse 74 elements All pairwise comparisons Mostly, arbitrary pairs of catalysts don’t even have any common elements. Can we meaningfully compare 'Na:33.2 Ti:0.5 Mn:66.3’ and 'Zn:77.8 Ce:22.2’ …? ['Na:33.2 Ti:0.5 Mn:66.3', 'Zn:77.8 Ce:22.2'] ['C:32.7 K:65.4 Pb:1.9', 'Y:100.0'] ['Na:66.7 Mo:33.3', 'Al:94.5 Mo:5.5'] ['C:4.0 Na:4.0 Ce:92.0', 'Y:70.0 Bi:30.0'] ['Si:98.2 Cs:1.8', 'Ti:50.0 Gd:50.0'] ['Na:9.1 Si:82.8 Cr:3.6 W:4.5', 'Na:20.0 Mg:80.0'] ['Y:66.7 Ba:33.3', 'Al:77.0 Ag:18.0 Ba:5.0'] ['Al:87.0 Cl:8.0 Fe:1.0 Sr:4.0', 'Na:33.2 Mn:66.3 Ta:0.5'] ['Na:1.0 La:99.0', 'Li:9.1 Ca:90.9'] ['Fe:100.0', 'Sr:50.0 Nd:50.0’] ['Al:75.0 Cl:16.0 Sr:8.0 Rh:1.0', 'Na:4.5 Si:79.2 Mn:16.3'] ['S:2.9 K:5.7 Ca:91.4', 'Sm:100.0'] ['P:34.5 Sr:65.5', 'Na:58.3 Cl:25.0 Mo:16.7'] ['Li:23.0 Si:73.2 W:3.8', 'Si:33.3 Ca:66.6 Pb:0.1'] ['Al:90.5 Ag:8.5 Pr:1.0', 'Na:66.7 Mo:33.3'] ['Cl:20.0 Ba:10.0 Nd:70.0', 'Mg:90.9 La:9.1'] ['Gd:100.0', 'Mn:50.0 Mo:50.0'] ['Na:76.9 Nb:23.1', 'La:90.0 Pb:10.0'] ['Li:6.5 S:3.2 Ca:90.3', 'P:34.5 Sr:65.5'] ['Na:5.0 Si:72.0 Cl:5.0 Mn:18.0', 'P:34.0 S:7.5 Ca:51.0 Pb:7.5']
32. / 35 20 Problem #3: It’s biased Unavoidable Human-Caused Biases “most chemical experiments are planned by human scientists and therefore are subject to a variety of human cognitive biases, heuristics and social influences.” Jia, X.; Lynch, A.; Huang, Y.; Danielson, M.; Lang’at, I.; Milder, A.; Ruby, A. E.; Wang, H.; Friedler, S. A.; Norquist, A. J.; Schrier, J. Nature 2019, 573 (7773), 251–255. Catalyst such as LaO3, Li/MgO, and Mn/Na2WO4/SiO2 extensively studied.
33. / 35 21 Our solutions Solution to Problem #1 (Underspecification) Tree ensemble regressors with prediction variances are used to make robust and less risky prediction as well as to quantify how uncertain each ML prediction is. Solution to Problem #2 (Sparsity) The catalyst representation called SWED (Sorted Weighted Elemental Descriptors) is developed to represent catalysts not in a one-hot fashion but by elemental descriptors. Solution to Problem #3 (Strong Bias) On the top of the above two, sequential model-based optimization with SWED only by 3 descriptors (electronegativity, density, and ΔHfus) as well as 8 descriptors are explored. Also, for suggested candidates to be worth testing, SHAP interpretations are provided. OCM Dataset Update, Reanalysis, Exploration The original dataset (1866 catalyst records from 421 reports until 2009) is extended to 4559 catalyst records from 542 reports from 2010 to 2019, and reanalyzed.
34. / 35 22 #1. Tree ensemble regression with uncertainty Tree ensemble regressors with prediction variances are used to make robust and less risky prediction as well as to quantify how uncertain each ML prediction is. Gradient Boosted Trees Extra Trees (no bootstrap) Random Forest Extra Trees (bootstrap) sample max sample min GradientBoostingRegressor LGBMRegressor RandomForestRegressor ExtraTreesRegressor By quantile regression to .16, .5, .84 quantiles Naturally by the law of total variance bounded prediction Avoid the risk of unintended extrapolation? (High-dimensional feature spaces can be counterintuitive…)
35. / 35 23 #2. SWED representation of catalysts The catalyst representation called SWED (Sorted Weighted Elemental Descriptors) is developed to represent catalysts not in a one-hot fashion but by elemental descriptors. Key Idea one-hot-like features below are statistically incomparable so represent catalysts instead by any elemental descriptors to represent arbitrary (used/unused) elements in a common ground, considering chemical similarities.
36. / 35 24 SWED (Sorted Weighted Elemental Descriptors) • Product terms can represent interaction effects between variables (e.g. probabilistic gating, attention, …) and furthermore, they can zero out the feature when the corresponding element is 0%. • Sorted concatenation is lossless, and was better than weighted sum or weighted max. • SWED lose the exact composition. To compensate, we also developed a SWED→composition estimator. • We tried many other things (matrix decomposition, Aitchison geometry, GNN, etc) that didn’t work. +
37. / 35 25 #3. Optimism in the face of uncertainty <latexit sha1_base64="Nr2ImZk9Gvp+/1cBWPwAK6yswp0=">AAACiXichVFNLwNRFD0dX1VfxUZi02iIVXNHBOmq0Y1lP7QkiMyMh9H5ysy0UU3/gJWdYEViIX6AH2DjD1j0J4gliY2F2+kkguBO3rzzzrvnvvPeVR1D93yiVkTq6u7p7Yv2xwYGh4ZH4qNjZc+uupooabZhu+uq4glDt0TJ131DrDuuUEzVEGtqJdveX6sJ19Nta9WvO2LLVPYsfVfXFJ+p8qZqNg6b2/EkpSiIxE8ghyCJMHJ2/A6b2IENDVWYELDgMzagwONvAzIIDnNbaDDnMtKDfYEmYqytcpbgDIXZCv/3eLURshav2zW9QK3xKQYPl5UJTNMj3dALPdAtPdH7r7UaQY22lzrPakcrnO2R44ni278qk2cf+5+qPz372MVS4FVn707AtG+hdfS1o9OXYrow3ZihK3pm/5fUonu+gVV71a7zonDxhx+VvfCLcYPk7+34CcpzKXkhRfn5ZGY5bFUUk5jCLPdjERmsIIcS1z/ACc5wLg1IsrQkpTupUiTUjONLSNkPIVKSSQ==</latexit> x <latexit sha1_base64="Qqfga3YaBoUI0Yb7fQVzumPP1Sw=">AAAChHichVHLSsNAFD1GrbW+qm4EN8WiuJBy6xsXUnTjsg9rC1VKEqc1mCYhSQu1+AO6VVy4UnAhfoAf4MYfcNFPEJcKblx4mwZEi3rDZM6cuefOmbmKpWuOS9Tskrp7egN9wf7QwODQ8Eh4dGzHMau2KrKqqZt2XpEdoWuGyLqaq4u8ZQu5ougipxxutvZzNWE7mmlsu3VL7FXksqGVNFV2mUrVi+EoxciLSCeI+yAKP5Jm+B672IcJFVVUIGDAZaxDhsNfAXEQLOb20GDOZqR5+wLHCLG2ylmCM2RmD/lf5lXBZw1et2o6nlrlU3QeNisjmKYnuqVXeqQ7eqaPX2s1vBotL3WelbZWWMWRk4nM+7+qCs8uDr5Uf3p2UcKq51Vj75bHtG6htvW1o4vXzFp6ujFD1/TC/q+oSQ98A6P2pt6kRPryDz8Ke+EX4wbFf7ajE+zMx+LLMUotRhMbfquCmMQUZrkfK0hgC0lkub7AKc5wLgWkOWlBWmqnSl2+ZhzfQlr/BNcbj/U=</latexit> y sample max sample min We would like to find X better than (hopefully) the currently known best . <latexit sha1_base64="g1U7Tzgl+Lj3TSOL2Mnl/fuxw90=">AAACi3ichVHLSsNAFL2Nr1qtrboR3BRLxVW5UVEpLooiuOzDPqAtJYljDc2LJC3W0B9w6cZF3Si4ED/AD3DjD7joJ4jLCm5ceJsGRIv1hsmcOXPPnTNzRUORLRux6+PGxicmp/zTgZnZ4FwoPL+Qt/SGKbGcpCu6WRQFiymyxnK2bCusaJhMUEWFFcT6fn+/0GSmJevakd0yWEUVapp8IkuCTVSxLKrOWbvKV8NRjKMbkWHAeyAKXqT08COU4Rh0kKABKjDQwCasgAAWfSXgAcEgrgIOcSYh2d1n0IYAaRuUxShDILZO/xqtSh6r0bpf03LVEp2i0DBJGYEYvuA99vAZH/AVP/+s5bg1+l5aNIsDLTOqoYul7Me/KpVmG06/VSM923ACO65XmbwbLtO/hTTQN8+vetlEJuas4i2+kf8b7OIT3UBrvkt3aZbpjPAjkhd6MWoQ/7sdwyC/Hue34pjejCb3vFb5YRlWYI36sQ1JOIQU5Nw+XEIHrrkgt8EluN1BKufzNIvwI7iDL6QMku0=</latexit> x1 <latexit sha1_base64="HrABz/ySZPgA1Ybj81k3XV8tH68=">AAACi3ichVHNSgJRFD5Of2aZVpugjSRGKzlaVEgLKYKW/uQPqMjMdLXB+WNmlGzwBVq2aWGbghbRA/QAbXqBFj5CtDRo06LjOBAl2Rnu3O9+93znfvceQZcl00LsebiJyanpGe+sb27evxAILi7lTa1piCwnarJmFAXeZLKkspwlWTIr6gbjFUFmBaFxMNgvtJhhSpp6bLV1VlH4uirVJJG3iCqWBcU+61Tj1WAYo+hEaBTEXBAGN1Ja8BHKcAIaiNAEBRioYBGWgQeTvhLEAEEnrgI2cQYhydln0AEfaZuUxSiDJ7ZB/zqtSi6r0npQ03TUIp0i0zBIGYIIvuA99vEZH/AVP/+sZTs1Bl7aNAtDLdOrgYuV7Me/KoVmC06/VWM9W1CDXcerRN51hxncQhzqW+dX/WwiE7HX8RbfyP8N9vCJbqC23sW7NMt0x/gRyAu9GDUo9rsdoyAfj8a2o5jeCif33VZ5YRXWYIP6sQNJOIIU5Jw+XEIXrjk/t8kluL1hKudxNcvwI7jDL6Ysku4=</latexit> x2 <latexit sha1_base64="qj6YHRPaHOekeYfY3235sFbWBxQ=">AAAChHichVHNLgNRFP6M/98WG4lNoyEW0pzpn9YCYWOpqDYpkZlxMel0ZjIzbULjBdgSCysSC/EAHsDGC1h4BLEksbFwZloRi3Ju7r3nfud85373HtU2dNcjem6T2js6u7p7evv6BwaHQuHhkU3XqjqayGuWYTlFVXGFoZsi7+meIYq2I5SKaoiCWl7244WacFzdMje8Q1tsV5R9U9/TNcVjKLewE45SLJtJUzIeoRhRJk5pdlIkZ+VsRGbEtyiatmqF77GFXVjQUEUFAiY89g0ocHmUIINgM7aNOmMOe3oQFzhGH3OrnCU4Q2G0zOs+n0pN1OSzX9MN2BrfYvB0mBnBJD3RLb3RI93RC322rFUPavhaDnlXG1xh74ROxtY//mVVePdw8MP6U7OHPWQCrTprtwPEf4XW4NeOLt7W59Ym61N0Ta+s/4qe6YFfYNbetZucWLv8Q4/KWvjHuEHfXYi0djbjMTkdS+SS0cWlZqt6MI4JTHM/ZrGIFawiz/UFTnGGc6lLmpESUqqRKrU1OaP4ZdL8F+zrkAQ=</latexit> ? <latexit sha1_base64="qj6YHRPaHOekeYfY3235sFbWBxQ=">AAAChHichVHNLgNRFP6M/98WG4lNoyEW0pzpn9YCYWOpqDYpkZlxMel0ZjIzbULjBdgSCysSC/EAHsDGC1h4BLEksbFwZloRi3Ju7r3nfud85373HtU2dNcjem6T2js6u7p7evv6BwaHQuHhkU3XqjqayGuWYTlFVXGFoZsi7+meIYq2I5SKaoiCWl7244WacFzdMje8Q1tsV5R9U9/TNcVjKLewE45SLJtJUzIeoRhRJk5pdlIkZ+VsRGbEtyiatmqF77GFXVjQUEUFAiY89g0ocHmUIINgM7aNOmMOe3oQFzhGH3OrnCU4Q2G0zOs+n0pN1OSzX9MN2BrfYvB0mBnBJD3RLb3RI93RC322rFUPavhaDnlXG1xh74ROxtY//mVVePdw8MP6U7OHPWQCrTprtwPEf4XW4NeOLt7W59Ym61N0Ta+s/4qe6YFfYNbetZucWLv8Q4/KWvjHuEHfXYi0djbjMTkdS+SS0cWlZqt6MI4JTHM/ZrGIFawiz/UFTnGGc6lLmpESUqqRKrU1OaP4ZdL8F+zrkAQ=</latexit> ? <latexit sha1_base64="qj6YHRPaHOekeYfY3235sFbWBxQ=">AAAChHichVHNLgNRFP6M/98WG4lNoyEW0pzpn9YCYWOpqDYpkZlxMel0ZjIzbULjBdgSCysSC/EAHsDGC1h4BLEksbFwZloRi3Ju7r3nfud85373HtU2dNcjem6T2js6u7p7evv6BwaHQuHhkU3XqjqayGuWYTlFVXGFoZsi7+meIYq2I5SKaoiCWl7244WacFzdMje8Q1tsV5R9U9/TNcVjKLewE45SLJtJUzIeoRhRJk5pdlIkZ+VsRGbEtyiatmqF77GFXVjQUEUFAiY89g0ocHmUIINgM7aNOmMOe3oQFzhGH3OrnCU4Q2G0zOs+n0pN1OSzX9MN2BrfYvB0mBnBJD3RLb3RI93RC322rFUPavhaDnlXG1xh74ROxtY//mVVePdw8MP6U7OHPWQCrTprtwPEf4XW4NeOLt7W59Ym61N0Ta+s/4qe6YFfYNbetZucWLv8Q4/KWvjHuEHfXYi0djbjMTkdS+SS0cWlZqt6MI4JTHM/ZrGIFawiz/UFTnGGc6lLmpESUqqRKrU1OaP4ZdL8F+zrkAQ=</latexit> ? <latexit sha1_base64="qj6YHRPaHOekeYfY3235sFbWBxQ=">AAAChHichVHNLgNRFP6M/98WG4lNoyEW0pzpn9YCYWOpqDYpkZlxMel0ZjIzbULjBdgSCysSC/EAHsDGC1h4BLEksbFwZloRi3Ju7r3nfud85373HtU2dNcjem6T2js6u7p7evv6BwaHQuHhkU3XqjqayGuWYTlFVXGFoZsi7+meIYq2I5SKaoiCWl7244WacFzdMje8Q1tsV5R9U9/TNcVjKLewE45SLJtJUzIeoRhRJk5pdlIkZ+VsRGbEtyiatmqF77GFXVjQUEUFAiY89g0ocHmUIINgM7aNOmMOe3oQFzhGH3OrnCU4Q2G0zOs+n0pN1OSzX9MN2BrfYvB0mBnBJD3RLb3RI93RC322rFUPavhaDnlXG1xh74ROxtY//mVVePdw8MP6U7OHPWQCrTprtwPEf4XW4NeOLt7W59Ym61N0Ta+s/4qe6YFfYNbetZucWLv8Q4/KWvjHuEHfXYi0djbjMTkdS+SS0cWlZqt6MI4JTHM/ZrGIFawiz/UFTnGGc6lLmpESUqqRKrU1OaP4ZdL8F+zrkAQ=</latexit> ? <latexit sha1_base64="qj6YHRPaHOekeYfY3235sFbWBxQ=">AAAChHichVHNLgNRFP6M/98WG4lNoyEW0pzpn9YCYWOpqDYpkZlxMel0ZjIzbULjBdgSCysSC/EAHsDGC1h4BLEksbFwZloRi3Ju7r3nfud85373HtU2dNcjem6T2js6u7p7evv6BwaHQuHhkU3XqjqayGuWYTlFVXGFoZsi7+meIYq2I5SKaoiCWl7244WacFzdMje8Q1tsV5R9U9/TNcVjKLewE45SLJtJUzIeoRhRJk5pdlIkZ+VsRGbEtyiatmqF77GFXVjQUEUFAiY89g0ocHmUIINgM7aNOmMOe3oQFzhGH3OrnCU4Q2G0zOs+n0pN1OSzX9MN2BrfYvB0mBnBJD3RLb3RI93RC322rFUPavhaDnlXG1xh74ROxtY//mVVePdw8MP6U7OHPWQCrTprtwPEf4XW4NeOLt7W59Ym61N0Ta+s/4qe6YFfYNbetZucWLv8Q4/KWvjHuEHfXYi0djbjMTkdS+SS0cWlZqt6MI4JTHM/ZrGIFawiz/UFTnGGc6lLmpESUqqRKrU1OaP4ZdL8F+zrkAQ=</latexit> ? What next location of is likely to give higher ? <latexit sha1_base64="GizvUUdXn2Kx+/exfuoNL7eSBEo=">AAACiXichVG7TgJBFD2sL8QHqI2JDZFgrMgsEh5WRBtLHgImYMzuOuDqvrK7EJX4A1Z2Rq00sTB+gB9g4w9Y8AnGEhMbCy8LxliodzM7Z87cc+fMXNnSVMdlrOMThoZHRsf844GJyanpYGhmtuyYTVvhJcXUTHtLlhyuqQYvuaqr8S3L5pIua7wiH6z39istbjuqaWy6Rxbf1qWGodZVRXKJKtdkvX14shOKsFg8kxRTK2EPJFKiB+JiOhMWY8yLCAaRM0MPqGEXJhQ0oYPDgEtYgwSHvipEMFjEbaNNnE1I9fY5ThAgbZOyOGVIxB7Qv0Gr6oA1aN2r6XhqhU7RaNikDCPKntkd67Inds9e2MevtdpejZ6XI5rlvpZbO8HT+eL7vyqdZhd736o/PbuoI+15Vcm75TG9Wyh9fev4vFtcLUTbS+yGvZL/a9Zhj3QDo/Wm3OZ54eoPPzJ5oRejBn11Ifw7KMdjYjK2kk9EsmuDVvmxgEUsUz9SyGIDOZSo/j7OcIFLYUIQhbSw2k8VfAPNHH6EsP4J1yOSpA==</latexit> x <latexit sha1_base64="fNJj/X2guXDQcypavSpjd2rZHL0=">AAAChHichVG7SgNBFD1ZXzG+ojaCzWJQLGSZTWISLUS0sczDqKAiu+skLu6L3U0gBn9AW8XCSsFC/AA/wMYfsMgniKWCjYV3NxGxUO8yO2fO3HPnzFzVMXTPZ6wVEbq6e3r7ov2xgcGh4ZH46NiGZ9dcjZc127DdLVXxuKFbvOzrvsG3HJcrpmrwTfVwNdjfrHPX021r3W84fNdUqpZe0TXFJ6rQ2IsnmJRcyMjZlBiCdFYOQVLOLYiyxMJIoBN5O36PHezDhoYaTHBY8AkbUODRtw0ZDA5xu2gS5xLSw32OY8RIW6MsThkKsYf0r9Jqu8NatA5qeqFao1MMGi4pRUyzJ3bLXtkju2PP7OPXWs2wRuClQbPa1nJnb+RkovT+r8qk2cfBt+pPzz4qyIVedfLuhExwC62trx9dvJYWi9PNGXbNXsj/FWuxB7qBVX/Tbgq8ePmHH5W80ItRg766IP4ONpKSnJFShXRieaXTqigmMYVZ6kcWy1hDHmWqz3GKM5wLvcKckBLm26lCpKMZx48Qlj4BjPuQUA==</latexit> y Exploitation Make the best decision given current information Exploration Gather more information by probing uncertain areas A fundamental choice: exploitation-exploration tradeoff <latexit sha1_base64="g1U7Tzgl+Lj3TSOL2Mnl/fuxw90=">AAACi3ichVHLSsNAFL2Nr1qtrboR3BRLxVW5UVEpLooiuOzDPqAtJYljDc2LJC3W0B9w6cZF3Si4ED/AD3DjD7joJ4jLCm5ceJsGRIv1hsmcOXPPnTNzRUORLRux6+PGxicmp/zTgZnZ4FwoPL+Qt/SGKbGcpCu6WRQFiymyxnK2bCusaJhMUEWFFcT6fn+/0GSmJevakd0yWEUVapp8IkuCTVSxLKrOWbvKV8NRjKMbkWHAeyAKXqT08COU4Rh0kKABKjDQwCasgAAWfSXgAcEgrgIOcSYh2d1n0IYAaRuUxShDILZO/xqtSh6r0bpf03LVEp2i0DBJGYEYvuA99vAZH/AVP/+s5bg1+l5aNIsDLTOqoYul7Me/KpVmG06/VSM923ACO65XmbwbLtO/hTTQN8+vetlEJuas4i2+kf8b7OIT3UBrvkt3aZbpjPAjkhd6MWoQ/7sdwyC/Hue34pjejCb3vFb5YRlWYI36sQ1JOIQU5Nw+XEIHrrkgt8EluN1BKufzNIvwI7iDL6QMku0=</latexit> x1 <latexit sha1_base64="HrABz/ySZPgA1Ybj81k3XV8tH68=">AAACi3ichVHNSgJRFD5Of2aZVpugjSRGKzlaVEgLKYKW/uQPqMjMdLXB+WNmlGzwBVq2aWGbghbRA/QAbXqBFj5CtDRo06LjOBAl2Rnu3O9+93znfvceQZcl00LsebiJyanpGe+sb27evxAILi7lTa1piCwnarJmFAXeZLKkspwlWTIr6gbjFUFmBaFxMNgvtJhhSpp6bLV1VlH4uirVJJG3iCqWBcU+61Tj1WAYo+hEaBTEXBAGN1Ja8BHKcAIaiNAEBRioYBGWgQeTvhLEAEEnrgI2cQYhydln0AEfaZuUxSiDJ7ZB/zqtSi6r0npQ03TUIp0i0zBIGYIIvuA99vEZH/AVP/+sZTs1Bl7aNAtDLdOrgYuV7Me/KoVmC06/VWM9W1CDXcerRN51hxncQhzqW+dX/WwiE7HX8RbfyP8N9vCJbqC23sW7NMt0x/gRyAu9GDUo9rsdoyAfj8a2o5jeCif33VZ5YRXWYIP6sQNJOIIU5Jw+XEIXrjk/t8kluL1hKudxNcvwI7jDL6Ysku4=</latexit> x2 exploitative choice explorative choice <latexit sha1_base64="TEyxI6Ffe+b+OEFRulxYJ0sUB8w=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBGBdk8IVxRXTjkoc8EkTS1gEbStu0hQSJP2DiVhauNHFh/AA/wI0/4IJPMC4xcePCS2lilIi3mc6ZM/fcOTNXMlTFshnreISR0bHxCe+kb2p6ZtYfmJvPWnrdlHlG1lXdzEuixVVF4xlbsVWeN0wu1iSV56Tqfm8/1+Cmpejaod00eLEmVjSlrMiiTVS6ebxWCoRYhDkRHARRF4TgRkIPPOIIJ9Aho44aODTYhFWIsOgrIAoGg7giWsSZhBRnn+McPtLWKYtThkhslf4VWhVcVqN1r6blqGU6RaVhkjKIMHth96zLntkDe2Wff9ZqOTV6Xpo0S30tN0r+i8X0x7+qGs02Tr9VQz3bKGPH8aqQd8NhereQ+/rGWbub3k2FWyvslr2R/xvWYU90A63xLt8leep6iB+JvNCLUYOiv9sxCLLrkeh2ZCO5GYrvua3yYgnLWKV+xBDHARLIUP0KLnGFtuAVIsKWEOunCh5Xs4AfIcS/AEWBkJQ=</latexit> y⇤ <latexit sha1_base64="ivVWRQUyNt8f9PjHjZnGRvohU5E=">AAAChnichVG5TsNAEH2YK9wBGiSaiCgIUURjbqgiaChDQg6JS7ZZgoVjW7YTKUT8ABItFFQgUSA+gA+g4Qco8gmIMkg0FEwcI6BImNXuzr6ZN/t2R7UN3fWIah1SZ1d3T2+or39gcGh4JDw6lnWtkqOJjGYZlpNXFVcYuikynu4ZIm87QimqhsipJxuNeK4sHFe3zG2vYou9olIw9SNdUzyG0pX92YNwlOLkW+SXs0jy6pIckQMkisCSVvgRuziEBQ0lFCFgwmPfgAKXxw5kEGzG9lBlzGFP9+MCZ+hnbomzBGcojJ7wWuDTToCafG7UdH22xrcYPB1mRhCjF7qnOj3TA73SZ8taVb9GQ0uFd7XJFfbByPlE+uNfVpF3D8c/rLaaPRxhxdeqs3bbRxqv0Jr88ulVPb2WilWn6ZbeWP8N1eiJX2CW37W7LZG6bqNHZS38Y9yg7y5EWjvZubi8FJ/fWogm1oNWhTCJKcxwP5aRwCaSyHD9Ai5wiSspJMWlRWm5mSp1BJxx/DEp8QVx95Cq</latexit> y⇤ Random choice (e.g. random design) or evenly spaced sampling (e.g. full factorial design) can also work for lower dimensional exploration.
38. / 35 25 #3. Optimism in the face of uncertainty <latexit sha1_base64="Nr2ImZk9Gvp+/1cBWPwAK6yswp0=">AAACiXichVFNLwNRFD0dX1VfxUZi02iIVXNHBOmq0Y1lP7QkiMyMh9H5ysy0UU3/gJWdYEViIX6AH2DjD1j0J4gliY2F2+kkguBO3rzzzrvnvvPeVR1D93yiVkTq6u7p7Yv2xwYGh4ZH4qNjZc+uupooabZhu+uq4glDt0TJ131DrDuuUEzVEGtqJdveX6sJ19Nta9WvO2LLVPYsfVfXFJ+p8qZqNg6b2/EkpSiIxE8ghyCJMHJ2/A6b2IENDVWYELDgMzagwONvAzIIDnNbaDDnMtKDfYEmYqytcpbgDIXZCv/3eLURshav2zW9QK3xKQYPl5UJTNMj3dALPdAtPdH7r7UaQY22lzrPakcrnO2R44ni278qk2cf+5+qPz372MVS4FVn707AtG+hdfS1o9OXYrow3ZihK3pm/5fUonu+gVV71a7zonDxhx+VvfCLcYPk7+34CcpzKXkhRfn5ZGY5bFUUk5jCLPdjERmsIIcS1z/ACc5wLg1IsrQkpTupUiTUjONLSNkPIVKSSQ==</latexit> x <latexit sha1_base64="Qqfga3YaBoUI0Yb7fQVzumPP1Sw=">AAAChHichVHLSsNAFD1GrbW+qm4EN8WiuJBy6xsXUnTjsg9rC1VKEqc1mCYhSQu1+AO6VVy4UnAhfoAf4MYfcNFPEJcKblx4mwZEi3rDZM6cuefOmbmKpWuOS9Tskrp7egN9wf7QwODQ8Eh4dGzHMau2KrKqqZt2XpEdoWuGyLqaq4u8ZQu5ougipxxutvZzNWE7mmlsu3VL7FXksqGVNFV2mUrVi+EoxciLSCeI+yAKP5Jm+B672IcJFVVUIGDAZaxDhsNfAXEQLOb20GDOZqR5+wLHCLG2ylmCM2RmD/lf5lXBZw1et2o6nlrlU3QeNisjmKYnuqVXeqQ7eqaPX2s1vBotL3WelbZWWMWRk4nM+7+qCs8uDr5Uf3p2UcKq51Vj75bHtG6htvW1o4vXzFp6ujFD1/TC/q+oSQ98A6P2pt6kRPryDz8Ke+EX4wbFf7ajE+zMx+LLMUotRhMbfquCmMQUZrkfK0hgC0lkub7AKc5wLgWkOWlBWmqnSl2+ZhzfQlr/BNcbj/U=</latexit> y sample max sample min We would like to find X better than (hopefully) the currently known best . <latexit sha1_base64="g1U7Tzgl+Lj3TSOL2Mnl/fuxw90=">AAACi3ichVHLSsNAFL2Nr1qtrboR3BRLxVW5UVEpLooiuOzDPqAtJYljDc2LJC3W0B9w6cZF3Si4ED/AD3DjD7joJ4jLCm5ceJsGRIv1hsmcOXPPnTNzRUORLRux6+PGxicmp/zTgZnZ4FwoPL+Qt/SGKbGcpCu6WRQFiymyxnK2bCusaJhMUEWFFcT6fn+/0GSmJevakd0yWEUVapp8IkuCTVSxLKrOWbvKV8NRjKMbkWHAeyAKXqT08COU4Rh0kKABKjDQwCasgAAWfSXgAcEgrgIOcSYh2d1n0IYAaRuUxShDILZO/xqtSh6r0bpf03LVEp2i0DBJGYEYvuA99vAZH/AVP/+s5bg1+l5aNIsDLTOqoYul7Me/KpVmG06/VSM923ACO65XmbwbLtO/hTTQN8+vetlEJuas4i2+kf8b7OIT3UBrvkt3aZbpjPAjkhd6MWoQ/7sdwyC/Hue34pjejCb3vFb5YRlWYI36sQ1JOIQU5Nw+XEIHrrkgt8EluN1BKufzNIvwI7iDL6QMku0=</latexit> x1 <latexit sha1_base64="HrABz/ySZPgA1Ybj81k3XV8tH68=">AAACi3ichVHNSgJRFD5Of2aZVpugjSRGKzlaVEgLKYKW/uQPqMjMdLXB+WNmlGzwBVq2aWGbghbRA/QAbXqBFj5CtDRo06LjOBAl2Rnu3O9+93znfvceQZcl00LsebiJyanpGe+sb27evxAILi7lTa1piCwnarJmFAXeZLKkspwlWTIr6gbjFUFmBaFxMNgvtJhhSpp6bLV1VlH4uirVJJG3iCqWBcU+61Tj1WAYo+hEaBTEXBAGN1Ja8BHKcAIaiNAEBRioYBGWgQeTvhLEAEEnrgI2cQYhydln0AEfaZuUxSiDJ7ZB/zqtSi6r0npQ03TUIp0i0zBIGYIIvuA99vEZH/AVP/+sZTs1Bl7aNAtDLdOrgYuV7Me/KoVmC06/VWM9W1CDXcerRN51hxncQhzqW+dX/WwiE7HX8RbfyP8N9vCJbqC23sW7NMt0x/gRyAu9GDUo9rsdoyAfj8a2o5jeCif33VZ5YRXWYIP6sQNJOIIU5Jw+XEIXrjk/t8kluL1hKudxNcvwI7jDL6Ysku4=</latexit> x2 <latexit sha1_base64="qj6YHRPaHOekeYfY3235sFbWBxQ=">AAAChHichVHNLgNRFP6M/98WG4lNoyEW0pzpn9YCYWOpqDYpkZlxMel0ZjIzbULjBdgSCysSC/EAHsDGC1h4BLEksbFwZloRi3Ju7r3nfud85373HtU2dNcjem6T2js6u7p7evv6BwaHQuHhkU3XqjqayGuWYTlFVXGFoZsi7+meIYq2I5SKaoiCWl7244WacFzdMje8Q1tsV5R9U9/TNcVjKLewE45SLJtJUzIeoRhRJk5pdlIkZ+VsRGbEtyiatmqF77GFXVjQUEUFAiY89g0ocHmUIINgM7aNOmMOe3oQFzhGH3OrnCU4Q2G0zOs+n0pN1OSzX9MN2BrfYvB0mBnBJD3RLb3RI93RC322rFUPavhaDnlXG1xh74ROxtY//mVVePdw8MP6U7OHPWQCrTprtwPEf4XW4NeOLt7W59Ym61N0Ta+s/4qe6YFfYNbetZucWLv8Q4/KWvjHuEHfXYi0djbjMTkdS+SS0cWlZqt6MI4JTHM/ZrGIFawiz/UFTnGGc6lLmpESUqqRKrU1OaP4ZdL8F+zrkAQ=</latexit> ? <latexit sha1_base64="qj6YHRPaHOekeYfY3235sFbWBxQ=">AAAChHichVHNLgNRFP6M/98WG4lNoyEW0pzpn9YCYWOpqDYpkZlxMel0ZjIzbULjBdgSCysSC/EAHsDGC1h4BLEksbFwZloRi3Ju7r3nfud85373HtU2dNcjem6T2js6u7p7evv6BwaHQuHhkU3XqjqayGuWYTlFVXGFoZsi7+meIYq2I5SKaoiCWl7244WacFzdMje8Q1tsV5R9U9/TNcVjKLewE45SLJtJUzIeoRhRJk5pdlIkZ+VsRGbEtyiatmqF77GFXVjQUEUFAiY89g0ocHmUIINgM7aNOmMOe3oQFzhGH3OrnCU4Q2G0zOs+n0pN1OSzX9MN2BrfYvB0mBnBJD3RLb3RI93RC322rFUPavhaDnlXG1xh74ROxtY//mVVePdw8MP6U7OHPWQCrTprtwPEf4XW4NeOLt7W59Ym61N0Ta+s/4qe6YFfYNbetZucWLv8Q4/KWvjHuEHfXYi0djbjMTkdS+SS0cWlZqt6MI4JTHM/ZrGIFawiz/UFTnGGc6lLmpESUqqRKrU1OaP4ZdL8F+zrkAQ=</latexit> ? <latexit sha1_base64="qj6YHRPaHOekeYfY3235sFbWBxQ=">AAAChHichVHNLgNRFP6M/98WG4lNoyEW0pzpn9YCYWOpqDYpkZlxMel0ZjIzbULjBdgSCysSC/EAHsDGC1h4BLEksbFwZloRi3Ju7r3nfud85373HtU2dNcjem6T2js6u7p7evv6BwaHQuHhkU3XqjqayGuWYTlFVXGFoZsi7+meIYq2I5SKaoiCWl7244WacFzdMje8Q1tsV5R9U9/TNcVjKLewE45SLJtJUzIeoRhRJk5pdlIkZ+VsRGbEtyiatmqF77GFXVjQUEUFAiY89g0ocHmUIINgM7aNOmMOe3oQFzhGH3OrnCU4Q2G0zOs+n0pN1OSzX9MN2BrfYvB0mBnBJD3RLb3RI93RC322rFUPavhaDnlXG1xh74ROxtY//mVVePdw8MP6U7OHPWQCrTprtwPEf4XW4NeOLt7W59Ym61N0Ta+s/4qe6YFfYNbetZucWLv8Q4/KWvjHuEHfXYi0djbjMTkdS+SS0cWlZqt6MI4JTHM/ZrGIFawiz/UFTnGGc6lLmpESUqqRKrU1OaP4ZdL8F+zrkAQ=</latexit> ? <latexit sha1_base64="qj6YHRPaHOekeYfY3235sFbWBxQ=">AAAChHichVHNLgNRFP6M/98WG4lNoyEW0pzpn9YCYWOpqDYpkZlxMel0ZjIzbULjBdgSCysSC/EAHsDGC1h4BLEksbFwZloRi3Ju7r3nfud85373HtU2dNcjem6T2js6u7p7evv6BwaHQuHhkU3XqjqayGuWYTlFVXGFoZsi7+meIYq2I5SKaoiCWl7244WacFzdMje8Q1tsV5R9U9/TNcVjKLewE45SLJtJUzIeoRhRJk5pdlIkZ+VsRGbEtyiatmqF77GFXVjQUEUFAiY89g0ocHmUIINgM7aNOmMOe3oQFzhGH3OrnCU4Q2G0zOs+n0pN1OSzX9MN2BrfYvB0mBnBJD3RLb3RI93RC322rFUPavhaDnlXG1xh74ROxtY//mVVePdw8MP6U7OHPWQCrTprtwPEf4XW4NeOLt7W59Ym61N0Ta+s/4qe6YFfYNbetZucWLv8Q4/KWvjHuEHfXYi0djbjMTkdS+SS0cWlZqt6MI4JTHM/ZrGIFawiz/UFTnGGc6lLmpESUqqRKrU1OaP4ZdL8F+zrkAQ=</latexit> ? <latexit sha1_base64="qj6YHRPaHOekeYfY3235sFbWBxQ=">AAAChHichVHNLgNRFP6M/98WG4lNoyEW0pzpn9YCYWOpqDYpkZlxMel0ZjIzbULjBdgSCysSC/EAHsDGC1h4BLEksbFwZloRi3Ju7r3nfud85373HtU2dNcjem6T2js6u7p7evv6BwaHQuHhkU3XqjqayGuWYTlFVXGFoZsi7+meIYq2I5SKaoiCWl7244WacFzdMje8Q1tsV5R9U9/TNcVjKLewE45SLJtJUzIeoRhRJk5pdlIkZ+VsRGbEtyiatmqF77GFXVjQUEUFAiY89g0ocHmUIINgM7aNOmMOe3oQFzhGH3OrnCU4Q2G0zOs+n0pN1OSzX9MN2BrfYvB0mBnBJD3RLb3RI93RC322rFUPavhaDnlXG1xh74ROxtY//mVVePdw8MP6U7OHPWQCrTprtwPEf4XW4NeOLt7W59Ym61N0Ta+s/4qe6YFfYNbetZucWLv8Q4/KWvjHuEHfXYi0djbjMTkdS+SS0cWlZqt6MI4JTHM/ZrGIFawiz/UFTnGGc6lLmpESUqqRKrU1OaP4ZdL8F+zrkAQ=</latexit> ? What next location of is likely to give higher ? <latexit sha1_base64="GizvUUdXn2Kx+/exfuoNL7eSBEo=">AAACiXichVG7TgJBFD2sL8QHqI2JDZFgrMgsEh5WRBtLHgImYMzuOuDqvrK7EJX4A1Z2Rq00sTB+gB9g4w9Y8AnGEhMbCy8LxliodzM7Z87cc+fMXNnSVMdlrOMThoZHRsf844GJyanpYGhmtuyYTVvhJcXUTHtLlhyuqQYvuaqr8S3L5pIua7wiH6z39istbjuqaWy6Rxbf1qWGodZVRXKJKtdkvX14shOKsFg8kxRTK2EPJFKiB+JiOhMWY8yLCAaRM0MPqGEXJhQ0oYPDgEtYgwSHvipEMFjEbaNNnE1I9fY5ThAgbZOyOGVIxB7Qv0Gr6oA1aN2r6XhqhU7RaNikDCPKntkd67Inds9e2MevtdpejZ6XI5rlvpZbO8HT+eL7vyqdZhd736o/PbuoI+15Vcm75TG9Wyh9fev4vFtcLUTbS+yGvZL/a9Zhj3QDo/Wm3OZ54eoPPzJ5oRejBn11Ifw7KMdjYjK2kk9EsmuDVvmxgEUsUz9SyGIDOZSo/j7OcIFLYUIQhbSw2k8VfAPNHH6EsP4J1yOSpA==</latexit> x <latexit sha1_base64="fNJj/X2guXDQcypavSpjd2rZHL0=">AAAChHichVG7SgNBFD1ZXzG+ojaCzWJQLGSZTWISLUS0sczDqKAiu+skLu6L3U0gBn9AW8XCSsFC/AA/wMYfsMgniKWCjYV3NxGxUO8yO2fO3HPnzFzVMXTPZ6wVEbq6e3r7ov2xgcGh4ZH46NiGZ9dcjZc127DdLVXxuKFbvOzrvsG3HJcrpmrwTfVwNdjfrHPX021r3W84fNdUqpZe0TXFJ6rQ2IsnmJRcyMjZlBiCdFYOQVLOLYiyxMJIoBN5O36PHezDhoYaTHBY8AkbUODRtw0ZDA5xu2gS5xLSw32OY8RIW6MsThkKsYf0r9Jqu8NatA5qeqFao1MMGi4pRUyzJ3bLXtkju2PP7OPXWs2wRuClQbPa1nJnb+RkovT+r8qk2cfBt+pPzz4qyIVedfLuhExwC62trx9dvJYWi9PNGXbNXsj/FWuxB7qBVX/Tbgq8ePmHH5W80ItRg766IP4ONpKSnJFShXRieaXTqigmMYVZ6kcWy1hDHmWqz3GKM5wLvcKckBLm26lCpKMZx48Qlj4BjPuQUA==</latexit> y Exploitation Make the best decision given current information Exploration Gather more information by probing uncertain areas A fundamental choice: exploitation-exploration tradeoff <latexit sha1_base64="g1U7Tzgl+Lj3TSOL2Mnl/fuxw90=">AAACi3ichVHLSsNAFL2Nr1qtrboR3BRLxVW5UVEpLooiuOzDPqAtJYljDc2LJC3W0B9w6cZF3Si4ED/AD3DjD7joJ4jLCm5ceJsGRIv1hsmcOXPPnTNzRUORLRux6+PGxicmp/zTgZnZ4FwoPL+Qt/SGKbGcpCu6WRQFiymyxnK2bCusaJhMUEWFFcT6fn+/0GSmJevakd0yWEUVapp8IkuCTVSxLKrOWbvKV8NRjKMbkWHAeyAKXqT08COU4Rh0kKABKjDQwCasgAAWfSXgAcEgrgIOcSYh2d1n0IYAaRuUxShDILZO/xqtSh6r0bpf03LVEp2i0DBJGYEYvuA99vAZH/AVP/+s5bg1+l5aNIsDLTOqoYul7Me/KpVmG06/VSM923ACO65XmbwbLtO/hTTQN8+vetlEJuas4i2+kf8b7OIT3UBrvkt3aZbpjPAjkhd6MWoQ/7sdwyC/Hue34pjejCb3vFb5YRlWYI36sQ1JOIQU5Nw+XEIHrrkgt8EluN1BKufzNIvwI7iDL6QMku0=</latexit> x1 <latexit sha1_base64="HrABz/ySZPgA1Ybj81k3XV8tH68=">AAACi3ichVHNSgJRFD5Of2aZVpugjSRGKzlaVEgLKYKW/uQPqMjMdLXB+WNmlGzwBVq2aWGbghbRA/QAbXqBFj5CtDRo06LjOBAl2Rnu3O9+93znfvceQZcl00LsebiJyanpGe+sb27evxAILi7lTa1piCwnarJmFAXeZLKkspwlWTIr6gbjFUFmBaFxMNgvtJhhSpp6bLV1VlH4uirVJJG3iCqWBcU+61Tj1WAYo+hEaBTEXBAGN1Ja8BHKcAIaiNAEBRioYBGWgQeTvhLEAEEnrgI2cQYhydln0AEfaZuUxSiDJ7ZB/zqtSi6r0npQ03TUIp0i0zBIGYIIvuA99vEZH/AVP/+sZTs1Bl7aNAtDLdOrgYuV7Me/KoVmC06/VWM9W1CDXcerRN51hxncQhzqW+dX/WwiE7HX8RbfyP8N9vCJbqC23sW7NMt0x/gRyAu9GDUo9rsdoyAfj8a2o5jeCif33VZ5YRXWYIP6sQNJOIIU5Jw+XEIXrjk/t8kluL1hKudxNcvwI7jDL6Ysku4=</latexit> x2 exploitative choice explorative choice <latexit sha1_base64="TEyxI6Ffe+b+OEFRulxYJ0sUB8w=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBGBdk8IVxRXTjkoc8EkTS1gEbStu0hQSJP2DiVhauNHFh/AA/wI0/4IJPMC4xcePCS2lilIi3mc6ZM/fcOTNXMlTFshnreISR0bHxCe+kb2p6ZtYfmJvPWnrdlHlG1lXdzEuixVVF4xlbsVWeN0wu1iSV56Tqfm8/1+Cmpejaod00eLEmVjSlrMiiTVS6ebxWCoRYhDkRHARRF4TgRkIPPOIIJ9Aho44aODTYhFWIsOgrIAoGg7giWsSZhBRnn+McPtLWKYtThkhslf4VWhVcVqN1r6blqGU6RaVhkjKIMHth96zLntkDe2Wff9ZqOTV6Xpo0S30tN0r+i8X0x7+qGs02Tr9VQz3bKGPH8aqQd8NhereQ+/rGWbub3k2FWyvslr2R/xvWYU90A63xLt8leep6iB+JvNCLUYOiv9sxCLLrkeh2ZCO5GYrvua3yYgnLWKV+xBDHARLIUP0KLnGFtuAVIsKWEOunCh5Xs4AfIcS/AEWBkJQ=</latexit> y⇤ <latexit sha1_base64="ivVWRQUyNt8f9PjHjZnGRvohU5E=">AAAChnichVG5TsNAEH2YK9wBGiSaiCgIUURjbqgiaChDQg6JS7ZZgoVjW7YTKUT8ABItFFQgUSA+gA+g4Qco8gmIMkg0FEwcI6BImNXuzr6ZN/t2R7UN3fWIah1SZ1d3T2+or39gcGh4JDw6lnWtkqOJjGYZlpNXFVcYuikynu4ZIm87QimqhsipJxuNeK4sHFe3zG2vYou9olIw9SNdUzyG0pX92YNwlOLkW+SXs0jy6pIckQMkisCSVvgRuziEBQ0lFCFgwmPfgAKXxw5kEGzG9lBlzGFP9+MCZ+hnbomzBGcojJ7wWuDTToCafG7UdH22xrcYPB1mRhCjF7qnOj3TA73SZ8taVb9GQ0uFd7XJFfbByPlE+uNfVpF3D8c/rLaaPRxhxdeqs3bbRxqv0Jr88ulVPb2WilWn6ZbeWP8N1eiJX2CW37W7LZG6bqNHZS38Y9yg7y5EWjvZubi8FJ/fWogm1oNWhTCJKcxwP5aRwCaSyHD9Ai5wiSspJMWlRWm5mSp1BJxx/DEp8QVx95Cq</latexit> y⇤ Random choice (e.g. random design) or evenly spaced sampling (e.g. full factorial design) can also work for lower dimensional exploration.
39. / 35 26 #3. Optimism in the face of uncertainty <latexit sha1_base64="Nr2ImZk9Gvp+/1cBWPwAK6yswp0=">AAACiXichVFNLwNRFD0dX1VfxUZi02iIVXNHBOmq0Y1lP7QkiMyMh9H5ysy0UU3/gJWdYEViIX6AH2DjD1j0J4gliY2F2+kkguBO3rzzzrvnvvPeVR1D93yiVkTq6u7p7Yv2xwYGh4ZH4qNjZc+uupooabZhu+uq4glDt0TJ131DrDuuUEzVEGtqJdveX6sJ19Nta9WvO2LLVPYsfVfXFJ+p8qZqNg6b2/EkpSiIxE8ghyCJMHJ2/A6b2IENDVWYELDgMzagwONvAzIIDnNbaDDnMtKDfYEmYqytcpbgDIXZCv/3eLURshav2zW9QK3xKQYPl5UJTNMj3dALPdAtPdH7r7UaQY22lzrPakcrnO2R44ni278qk2cf+5+qPz372MVS4FVn707AtG+hdfS1o9OXYrow3ZihK3pm/5fUonu+gVV71a7zonDxhx+VvfCLcYPk7+34CcpzKXkhRfn5ZGY5bFUUk5jCLPdjERmsIIcS1z/ACc5wLg1IsrQkpTupUiTUjONLSNkPIVKSSQ==</latexit> x <latexit sha1_base64="Qqfga3YaBoUI0Yb7fQVzumPP1Sw=">AAAChHichVHLSsNAFD1GrbW+qm4EN8WiuJBy6xsXUnTjsg9rC1VKEqc1mCYhSQu1+AO6VVy4UnAhfoAf4MYfcNFPEJcKblx4mwZEi3rDZM6cuefOmbmKpWuOS9Tskrp7egN9wf7QwODQ8Eh4dGzHMau2KrKqqZt2XpEdoWuGyLqaq4u8ZQu5ougipxxutvZzNWE7mmlsu3VL7FXksqGVNFV2mUrVi+EoxciLSCeI+yAKP5Jm+B672IcJFVVUIGDAZaxDhsNfAXEQLOb20GDOZqR5+wLHCLG2ylmCM2RmD/lf5lXBZw1et2o6nlrlU3QeNisjmKYnuqVXeqQ7eqaPX2s1vBotL3WelbZWWMWRk4nM+7+qCs8uDr5Uf3p2UcKq51Vj75bHtG6htvW1o4vXzFp6ujFD1/TC/q+oSQ98A6P2pt6kRPryDz8Ke+EX4wbFf7ajE+zMx+LLMUotRhMbfquCmMQUZrkfK0hgC0lkub7AKc5wLgWkOWlBWmqnSl2+ZhzfQlr/BNcbj/U=</latexit> y sample max sample min ML fits a function to minimize the average errors, and as a result, ML functions go through the center (mean) of sample output values. the currently known best We would like to find X better than (hopefully) the currently known best. Now we would like to use ML for the goal. <latexit sha1_base64="TEyxI6Ffe+b+OEFRulxYJ0sUB8w=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBGBdk8IVxRXTjkoc8EkTS1gEbStu0hQSJP2DiVhauNHFh/AA/wI0/4IJPMC4xcePCS2lilIi3mc6ZM/fcOTNXMlTFshnreISR0bHxCe+kb2p6ZtYfmJvPWnrdlHlG1lXdzEuixVVF4xlbsVWeN0wu1iSV56Tqfm8/1+Cmpejaod00eLEmVjSlrMiiTVS6ebxWCoRYhDkRHARRF4TgRkIPPOIIJ9Aho44aODTYhFWIsOgrIAoGg7giWsSZhBRnn+McPtLWKYtThkhslf4VWhVcVqN1r6blqGU6RaVhkjKIMHth96zLntkDe2Wff9ZqOTV6Xpo0S30tN0r+i8X0x7+qGs02Tr9VQz3bKGPH8aqQd8NhereQ+/rGWbub3k2FWyvslr2R/xvWYU90A63xLt8leep6iB+JvNCLUYOiv9sxCLLrkeh2ZCO5GYrvua3yYgnLWKV+xBDHARLIUP0KLnGFtuAVIsKWEOunCh5Xs4AfIcS/AEWBkJQ=</latexit> y⇤ goes through here predictions will be groundless when it goes beyond this area unintended extrapolation When ML is rightly fitted, the predicted values are never larger than the known best, which is inconsistent with the goal.
40. / 35 26 #3. Optimism in the face of uncertainty <latexit sha1_base64="Nr2ImZk9Gvp+/1cBWPwAK6yswp0=">AAACiXichVFNLwNRFD0dX1VfxUZi02iIVXNHBOmq0Y1lP7QkiMyMh9H5ysy0UU3/gJWdYEViIX6AH2DjD1j0J4gliY2F2+kkguBO3rzzzrvnvvPeVR1D93yiVkTq6u7p7Yv2xwYGh4ZH4qNjZc+uupooabZhu+uq4glDt0TJ131DrDuuUEzVEGtqJdveX6sJ19Nta9WvO2LLVPYsfVfXFJ+p8qZqNg6b2/EkpSiIxE8ghyCJMHJ2/A6b2IENDVWYELDgMzagwONvAzIIDnNbaDDnMtKDfYEmYqytcpbgDIXZCv/3eLURshav2zW9QK3xKQYPl5UJTNMj3dALPdAtPdH7r7UaQY22lzrPakcrnO2R44ni278qk2cf+5+qPz372MVS4FVn707AtG+hdfS1o9OXYrow3ZihK3pm/5fUonu+gVV71a7zonDxhx+VvfCLcYPk7+34CcpzKXkhRfn5ZGY5bFUUk5jCLPdjERmsIIcS1z/ACc5wLg1IsrQkpTupUiTUjONLSNkPIVKSSQ==</latexit> x <latexit sha1_base64="g1U7Tzgl+Lj3TSOL2Mnl/fuxw90=">AAACi3ichVHLSsNAFL2Nr1qtrboR3BRLxVW5UVEpLooiuOzDPqAtJYljDc2LJC3W0B9w6cZF3Si4ED/AD3DjD7joJ4jLCm5ceJsGRIv1hsmcOXPPnTNzRUORLRux6+PGxicmp/zTgZnZ4FwoPL+Qt/SGKbGcpCu6WRQFiymyxnK2bCusaJhMUEWFFcT6fn+/0GSmJevakd0yWEUVapp8IkuCTVSxLKrOWbvKV8NRjKMbkWHAeyAKXqT08COU4Rh0kKABKjDQwCasgAAWfSXgAcEgrgIOcSYh2d1n0IYAaRuUxShDILZO/xqtSh6r0bpf03LVEp2i0DBJGYEYvuA99vAZH/AVP/+s5bg1+l5aNIsDLTOqoYul7Me/KpVmG06/VSM923ACO65XmbwbLtO/hTTQN8+vetlEJuas4i2+kf8b7OIT3UBrvkt3aZbpjPAjkhd6MWoQ/7sdwyC/Hue34pjejCb3vFb5YRlWYI36sQ1JOIQU5Nw+XEIHrrkgt8EluN1BKufzNIvwI7iDL6QMku0=</latexit> x1 <latexit sha1_base64="HrABz/ySZPgA1Ybj81k3XV8tH68=">AAACi3ichVHNSgJRFD5Of2aZVpugjSRGKzlaVEgLKYKW/uQPqMjMdLXB+WNmlGzwBVq2aWGbghbRA/QAbXqBFj5CtDRo06LjOBAl2Rnu3O9+93znfvceQZcl00LsebiJyanpGe+sb27evxAILi7lTa1piCwnarJmFAXeZLKkspwlWTIr6gbjFUFmBaFxMNgvtJhhSpp6bLV1VlH4uirVJJG3iCqWBcU+61Tj1WAYo+hEaBTEXBAGN1Ja8BHKcAIaiNAEBRioYBGWgQeTvhLEAEEnrgI2cQYhydln0AEfaZuUxSiDJ7ZB/zqtSi6r0npQ03TUIp0i0zBIGYIIvuA99vEZH/AVP/+sZTs1Bl7aNAtDLdOrgYuV7Me/KoVmC06/VWM9W1CDXcerRN51hxncQhzqW+dX/WwiE7HX8RbfyP8N9vCJbqC23sW7NMt0x/gRyAu9GDUo9rsdoyAfj8a2o5jeCif33VZ5YRXWYIP6sQNJOIIU5Jw+XEIXrjk/t8kluL1hKudxNcvwI7jDL6Ysku4=</latexit> x2 <latexit sha1_base64="Qqfga3YaBoUI0Yb7fQVzumPP1Sw=">AAAChHichVHLSsNAFD1GrbW+qm4EN8WiuJBy6xsXUnTjsg9rC1VKEqc1mCYhSQu1+AO6VVy4UnAhfoAf4MYfcNFPEJcKblx4mwZEi3rDZM6cuefOmbmKpWuOS9Tskrp7egN9wf7QwODQ8Eh4dGzHMau2KrKqqZt2XpEdoWuGyLqaq4u8ZQu5ougipxxutvZzNWE7mmlsu3VL7FXksqGVNFV2mUrVi+EoxciLSCeI+yAKP5Jm+B672IcJFVVUIGDAZaxDhsNfAXEQLOb20GDOZqR5+wLHCLG2ylmCM2RmD/lf5lXBZw1et2o6nlrlU3QeNisjmKYnuqVXeqQ7eqaPX2s1vBotL3WelbZWWMWRk4nM+7+qCs8uDr5Uf3p2UcKq51Vj75bHtG6htvW1o4vXzFp6ujFD1/TC/q+oSQ98A6P2pt6kRPryDz8Ke+EX4wbFf7ajE+zMx+LLMUotRhMbfquCmMQUZrkfK0hgC0lkub7AKc5wLgWkOWlBWmqnSl2+ZhzfQlr/BNcbj/U=</latexit> y sample max sample min ML fits a function to minimize the average errors, and as a result, ML functions go through the center (mean) of sample output values. the currently known best We would like to find X better than (hopefully) the currently known best. Now we would like to use ML for the goal. <latexit sha1_base64="TEyxI6Ffe+b+OEFRulxYJ0sUB8w=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBGBdk8IVxRXTjkoc8EkTS1gEbStu0hQSJP2DiVhauNHFh/AA/wI0/4IJPMC4xcePCS2lilIi3mc6ZM/fcOTNXMlTFshnreISR0bHxCe+kb2p6ZtYfmJvPWnrdlHlG1lXdzEuixVVF4xlbsVWeN0wu1iSV56Tqfm8/1+Cmpejaod00eLEmVjSlrMiiTVS6ebxWCoRYhDkRHARRF4TgRkIPPOIIJ9Aho44aODTYhFWIsOgrIAoGg7giWsSZhBRnn+McPtLWKYtThkhslf4VWhVcVqN1r6blqGU6RaVhkjKIMHth96zLntkDe2Wff9ZqOTV6Xpo0S30tN0r+i8X0x7+qGs02Tr9VQz3bKGPH8aqQd8NhereQ+/rGWbub3k2FWyvslr2R/xvWYU90A63xLt8leep6iB+JvNCLUYOiv9sxCLLrkeh2ZCO5GYrvua3yYgnLWKV+xBDHARLIUP0KLnGFtuAVIsKWEOunCh5Xs4AfIcS/AEWBkJQ=</latexit> y⇤ goes through here predictions will be groundless when it goes beyond this area unintended extrapolation When ML is rightly fitted, the predicted values are never larger than the known best, which is inconsistent with the goal.
41. / 35 27 #3. Optimism in the face of uncertainty <latexit sha1_base64="Nr2ImZk9Gvp+/1cBWPwAK6yswp0=">AAACiXichVFNLwNRFD0dX1VfxUZi02iIVXNHBOmq0Y1lP7QkiMyMh9H5ysy0UU3/gJWdYEViIX6AH2DjD1j0J4gliY2F2+kkguBO3rzzzrvnvvPeVR1D93yiVkTq6u7p7Yv2xwYGh4ZH4qNjZc+uupooabZhu+uq4glDt0TJ131DrDuuUEzVEGtqJdveX6sJ19Nta9WvO2LLVPYsfVfXFJ+p8qZqNg6b2/EkpSiIxE8ghyCJMHJ2/A6b2IENDVWYELDgMzagwONvAzIIDnNbaDDnMtKDfYEmYqytcpbgDIXZCv/3eLURshav2zW9QK3xKQYPl5UJTNMj3dALPdAtPdH7r7UaQY22lzrPakcrnO2R44ni278qk2cf+5+qPz372MVS4FVn707AtG+hdfS1o9OXYrow3ZihK3pm/5fUonu+gVV71a7zonDxhx+VvfCLcYPk7+34CcpzKXkhRfn5ZGY5bFUUk5jCLPdjERmsIIcS1z/ACc5wLg1IsrQkpTupUiTUjONLSNkPIVKSSQ==</latexit> x <latexit sha1_base64="g1U7Tzgl+Lj3TSOL2Mnl/fuxw90=">AAACi3ichVHLSsNAFL2Nr1qtrboR3BRLxVW5UVEpLooiuOzDPqAtJYljDc2LJC3W0B9w6cZF3Si4ED/AD3DjD7joJ4jLCm5ceJsGRIv1hsmcOXPPnTNzRUORLRux6+PGxicmp/zTgZnZ4FwoPL+Qt/SGKbGcpCu6WRQFiymyxnK2bCusaJhMUEWFFcT6fn+/0GSmJevakd0yWEUVapp8IkuCTVSxLKrOWbvKV8NRjKMbkWHAeyAKXqT08COU4Rh0kKABKjDQwCasgAAWfSXgAcEgrgIOcSYh2d1n0IYAaRuUxShDILZO/xqtSh6r0bpf03LVEp2i0DBJGYEYvuA99vAZH/AVP/+s5bg1+l5aNIsDLTOqoYul7Me/KpVmG06/VSM923ACO65XmbwbLtO/hTTQN8+vetlEJuas4i2+kf8b7OIT3UBrvkt3aZbpjPAjkhd6MWoQ/7sdwyC/Hue34pjejCb3vFb5YRlWYI36sQ1JOIQU5Nw+XEIHrrkgt8EluN1BKufzNIvwI7iDL6QMku0=</latexit> x1 <latexit sha1_base64="HrABz/ySZPgA1Ybj81k3XV8tH68=">AAACi3ichVHNSgJRFD5Of2aZVpugjSRGKzlaVEgLKYKW/uQPqMjMdLXB+WNmlGzwBVq2aWGbghbRA/QAbXqBFj5CtDRo06LjOBAl2Rnu3O9+93znfvceQZcl00LsebiJyanpGe+sb27evxAILi7lTa1piCwnarJmFAXeZLKkspwlWTIr6gbjFUFmBaFxMNgvtJhhSpp6bLV1VlH4uirVJJG3iCqWBcU+61Tj1WAYo+hEaBTEXBAGN1Ja8BHKcAIaiNAEBRioYBGWgQeTvhLEAEEnrgI2cQYhydln0AEfaZuUxSiDJ7ZB/zqtSi6r0npQ03TUIp0i0zBIGYIIvuA99vEZH/AVP/+sZTs1Bl7aNAtDLdOrgYuV7Me/KoVmC06/VWM9W1CDXcerRN51hxncQhzqW+dX/WwiE7HX8RbfyP8N9vCJbqC23sW7NMt0x/gRyAu9GDUo9rsdoyAfj8a2o5jeCif33VZ5YRXWYIP6sQNJOIIU5Jw+XEIXrjk/t8kluL1hKudxNcvwI7jDL6Ysku4=</latexit> x2 <latexit sha1_base64="Qqfga3YaBoUI0Yb7fQVzumPP1Sw=">AAAChHichVHLSsNAFD1GrbW+qm4EN8WiuJBy6xsXUnTjsg9rC1VKEqc1mCYhSQu1+AO6VVy4UnAhfoAf4MYfcNFPEJcKblx4mwZEi3rDZM6cuefOmbmKpWuOS9Tskrp7egN9wf7QwODQ8Eh4dGzHMau2KrKqqZt2XpEdoWuGyLqaq4u8ZQu5ougipxxutvZzNWE7mmlsu3VL7FXksqGVNFV2mUrVi+EoxciLSCeI+yAKP5Jm+B672IcJFVVUIGDAZaxDhsNfAXEQLOb20GDOZqR5+wLHCLG2ylmCM2RmD/lf5lXBZw1et2o6nlrlU3QeNisjmKYnuqVXeqQ7eqaPX2s1vBotL3WelbZWWMWRk4nM+7+qCs8uDr5Uf3p2UcKq51Vj75bHtG6htvW1o4vXzFp6ujFD1/TC/q+oSQ98A6P2pt6kRPryDz8Ke+EX4wbFf7ajE+zMx+LLMUotRhMbfquCmMQUZrkfK0hgC0lkub7AKc5wLgWkOWlBWmqnSl2+ZhzfQlr/BNcbj/U=</latexit> y sample max sample min This is why we need a criterion taking uncertainty into consideration instead of direct use of ML predicted values to guide exploration. It’ll be nice to gather more information around here even though the mean is not so high (the predictions have a large variance) <latexit sha1_base64="i+Yum6tkJlYfyXr5MYELEoyKzkM=">AAAChHichVFNLwNRFD3GV3222EhsGk3FQpo7rZZaSMPGUlGVIDIzHiadzkxmpk2q8QfYEgsrEgvxA/wAG3/Awk8QSxIbC3dmKmKBO3nzzjvvnvvOe1e1Dd31iJ7apPaOzq7uSE9vX//AYDQ2NLzuWjVHEyXNMixnQ1VcYeimKHm6Z4gN2xFKVTVEWa0s+vvlunBc3TLXvIYttqvKvqnv6ZriMVVs7MQSlErnc2nKx0OQzYVAzqfjcoqCSKAVy1bsDlvYhQUNNVQhYMJjbECBy98mZBBs5rbRZM5hpAf7AkfoZW2NswRnKMxW+L/Pq80Wa/Lar+kGao1PMXg4rIwjSY90Q6/0QLf0TB+/1moGNXwvDZ7VUCvsnejx6Or7v6oqzx4OvlV/evawh9nAq87e7YDxb6GF+vrh+evq3EqyOUFX9ML+L+mJ7vkGZv1Nuy6KlYs//KjshV+MG/TVhfjvYD2dknOpTHE6UVhotSqCMYxjkvsxgwKWsIwS1xc4wSnOpC5pSspI2TBVamtpRvAjpPlPiwOQTw==</latexit> y
42. / 35 27 #3. Optimism in the face of uncertainty <latexit sha1_base64="Nr2ImZk9Gvp+/1cBWPwAK6yswp0=">AAACiXichVFNLwNRFD0dX1VfxUZi02iIVXNHBOmq0Y1lP7QkiMyMh9H5ysy0UU3/gJWdYEViIX6AH2DjD1j0J4gliY2F2+kkguBO3rzzzrvnvvPeVR1D93yiVkTq6u7p7Yv2xwYGh4ZH4qNjZc+uupooabZhu+uq4glDt0TJ131DrDuuUEzVEGtqJdveX6sJ19Nta9WvO2LLVPYsfVfXFJ+p8qZqNg6b2/EkpSiIxE8ghyCJMHJ2/A6b2IENDVWYELDgMzagwONvAzIIDnNbaDDnMtKDfYEmYqytcpbgDIXZCv/3eLURshav2zW9QK3xKQYPl5UJTNMj3dALPdAtPdH7r7UaQY22lzrPakcrnO2R44ni278qk2cf+5+qPz372MVS4FVn707AtG+hdfS1o9OXYrow3ZihK3pm/5fUonu+gVV71a7zonDxhx+VvfCLcYPk7+34CcpzKXkhRfn5ZGY5bFUUk5jCLPdjERmsIIcS1z/ACc5wLg1IsrQkpTupUiTUjONLSNkPIVKSSQ==</latexit> x <latexit sha1_base64="g1U7Tzgl+Lj3TSOL2Mnl/fuxw90=">AAACi3ichVHLSsNAFL2Nr1qtrboR3BRLxVW5UVEpLooiuOzDPqAtJYljDc2LJC3W0B9w6cZF3Si4ED/AD3DjD7joJ4jLCm5ceJsGRIv1hsmcOXPPnTNzRUORLRux6+PGxicmp/zTgZnZ4FwoPL+Qt/SGKbGcpCu6WRQFiymyxnK2bCusaJhMUEWFFcT6fn+/0GSmJevakd0yWEUVapp8IkuCTVSxLKrOWbvKV8NRjKMbkWHAeyAKXqT08COU4Rh0kKABKjDQwCasgAAWfSXgAcEgrgIOcSYh2d1n0IYAaRuUxShDILZO/xqtSh6r0bpf03LVEp2i0DBJGYEYvuA99vAZH/AVP/+s5bg1+l5aNIsDLTOqoYul7Me/KpVmG06/VSM923ACO65XmbwbLtO/hTTQN8+vetlEJuas4i2+kf8b7OIT3UBrvkt3aZbpjPAjkhd6MWoQ/7sdwyC/Hue34pjejCb3vFb5YRlWYI36sQ1JOIQU5Nw+XEIHrrkgt8EluN1BKufzNIvwI7iDL6QMku0=</latexit> x1 <latexit sha1_base64="HrABz/ySZPgA1Ybj81k3XV8tH68=">AAACi3ichVHNSgJRFD5Of2aZVpugjSRGKzlaVEgLKYKW/uQPqMjMdLXB+WNmlGzwBVq2aWGbghbRA/QAbXqBFj5CtDRo06LjOBAl2Rnu3O9+93znfvceQZcl00LsebiJyanpGe+sb27evxAILi7lTa1piCwnarJmFAXeZLKkspwlWTIr6gbjFUFmBaFxMNgvtJhhSpp6bLV1VlH4uirVJJG3iCqWBcU+61Tj1WAYo+hEaBTEXBAGN1Ja8BHKcAIaiNAEBRioYBGWgQeTvhLEAEEnrgI2cQYhydln0AEfaZuUxSiDJ7ZB/zqtSi6r0npQ03TUIp0i0zBIGYIIvuA99vEZH/AVP/+sZTs1Bl7aNAtDLdOrgYuV7Me/KoVmC06/VWM9W1CDXcerRN51hxncQhzqW+dX/WwiE7HX8RbfyP8N9vCJbqC23sW7NMt0x/gRyAu9GDUo9rsdoyAfj8a2o5jeCif33VZ5YRXWYIP6sQNJOIIU5Jw+XEIXrjk/t8kluL1hKudxNcvwI7jDL6Ysku4=</latexit> x2 <latexit sha1_base64="Qqfga3YaBoUI0Yb7fQVzumPP1Sw=">AAAChHichVHLSsNAFD1GrbW+qm4EN8WiuJBy6xsXUnTjsg9rC1VKEqc1mCYhSQu1+AO6VVy4UnAhfoAf4MYfcNFPEJcKblx4mwZEi3rDZM6cuefOmbmKpWuOS9Tskrp7egN9wf7QwODQ8Eh4dGzHMau2KrKqqZt2XpEdoWuGyLqaq4u8ZQu5ougipxxutvZzNWE7mmlsu3VL7FXksqGVNFV2mUrVi+EoxciLSCeI+yAKP5Jm+B672IcJFVVUIGDAZaxDhsNfAXEQLOb20GDOZqR5+wLHCLG2ylmCM2RmD/lf5lXBZw1et2o6nlrlU3QeNisjmKYnuqVXeqQ7eqaPX2s1vBotL3WelbZWWMWRk4nM+7+qCs8uDr5Uf3p2UcKq51Vj75bHtG6htvW1o4vXzFp6ujFD1/TC/q+oSQ98A6P2pt6kRPryDz8Ke+EX4wbFf7ajE+zMx+LLMUotRhMbfquCmMQUZrkfK0hgC0lkub7AKc5wLgWkOWlBWmqnSl2+ZhzfQlr/BNcbj/U=</latexit> y sample max sample min This is why we need a criterion taking uncertainty into consideration instead of direct use of ML predicted values to guide exploration. It’ll be nice to gather more information around here even though the mean is not so high (the predictions have a large variance) <latexit sha1_base64="i+Yum6tkJlYfyXr5MYELEoyKzkM=">AAAChHichVFNLwNRFD3GV3222EhsGk3FQpo7rZZaSMPGUlGVIDIzHiadzkxmpk2q8QfYEgsrEgvxA/wAG3/Awk8QSxIbC3dmKmKBO3nzzjvvnvvOe1e1Dd31iJ7apPaOzq7uSE9vX//AYDQ2NLzuWjVHEyXNMixnQ1VcYeimKHm6Z4gN2xFKVTVEWa0s+vvlunBc3TLXvIYttqvKvqnv6ZriMVVs7MQSlErnc2nKx0OQzYVAzqfjcoqCSKAVy1bsDlvYhQUNNVQhYMJjbECBy98mZBBs5rbRZM5hpAf7AkfoZW2NswRnKMxW+L/Pq80Wa/Lar+kGao1PMXg4rIwjSY90Q6/0QLf0TB+/1moGNXwvDZ7VUCvsnejx6Or7v6oqzx4OvlV/evawh9nAq87e7YDxb6GF+vrh+evq3EqyOUFX9ML+L+mJ7vkGZv1Nuy6KlYs//KjshV+MG/TVhfjvYD2dknOpTHE6UVhotSqCMYxjkvsxgwKWsIwS1xc4wSnOpC5pSspI2TBVamtpRvAjpPlPiwOQTw==</latexit> y
43. / 35 27 #3. Optimism in the face of uncertainty <latexit sha1_base64="Nr2ImZk9Gvp+/1cBWPwAK6yswp0=">AAACiXichVFNLwNRFD0dX1VfxUZi02iIVXNHBOmq0Y1lP7QkiMyMh9H5ysy0UU3/gJWdYEViIX6AH2DjD1j0J4gliY2F2+kkguBO3rzzzrvnvvPeVR1D93yiVkTq6u7p7Yv2xwYGh4ZH4qNjZc+uupooabZhu+uq4glDt0TJ131DrDuuUEzVEGtqJdveX6sJ19Nta9WvO2LLVPYsfVfXFJ+p8qZqNg6b2/EkpSiIxE8ghyCJMHJ2/A6b2IENDVWYELDgMzagwONvAzIIDnNbaDDnMtKDfYEmYqytcpbgDIXZCv/3eLURshav2zW9QK3xKQYPl5UJTNMj3dALPdAtPdH7r7UaQY22lzrPakcrnO2R44ni278qk2cf+5+qPz372MVS4FVn707AtG+hdfS1o9OXYrow3ZihK3pm/5fUonu+gVV71a7zonDxhx+VvfCLcYPk7+34CcpzKXkhRfn5ZGY5bFUUk5jCLPdjERmsIIcS1z/ACc5wLg1IsrQkpTupUiTUjONLSNkPIVKSSQ==</latexit> x <latexit sha1_base64="g1U7Tzgl+Lj3TSOL2Mnl/fuxw90=">AAACi3ichVHLSsNAFL2Nr1qtrboR3BRLxVW5UVEpLooiuOzDPqAtJYljDc2LJC3W0B9w6cZF3Si4ED/AD3DjD7joJ4jLCm5ceJsGRIv1hsmcOXPPnTNzRUORLRux6+PGxicmp/zTgZnZ4FwoPL+Qt/SGKbGcpCu6WRQFiymyxnK2bCusaJhMUEWFFcT6fn+/0GSmJevakd0yWEUVapp8IkuCTVSxLKrOWbvKV8NRjKMbkWHAeyAKXqT08COU4Rh0kKABKjDQwCasgAAWfSXgAcEgrgIOcSYh2d1n0IYAaRuUxShDILZO/xqtSh6r0bpf03LVEp2i0DBJGYEYvuA99vAZH/AVP/+s5bg1+l5aNIsDLTOqoYul7Me/KpVmG06/VSM923ACO65XmbwbLtO/hTTQN8+vetlEJuas4i2+kf8b7OIT3UBrvkt3aZbpjPAjkhd6MWoQ/7sdwyC/Hue34pjejCb3vFb5YRlWYI36sQ1JOIQU5Nw+XEIHrrkgt8EluN1BKufzNIvwI7iDL6QMku0=</latexit> x1 <latexit sha1_base64="HrABz/ySZPgA1Ybj81k3XV8tH68=">AAACi3ichVHNSgJRFD5Of2aZVpugjSRGKzlaVEgLKYKW/uQPqMjMdLXB+WNmlGzwBVq2aWGbghbRA/QAbXqBFj5CtDRo06LjOBAl2Rnu3O9+93znfvceQZcl00LsebiJyanpGe+sb27evxAILi7lTa1piCwnarJmFAXeZLKkspwlWTIr6gbjFUFmBaFxMNgvtJhhSpp6bLV1VlH4uirVJJG3iCqWBcU+61Tj1WAYo+hEaBTEXBAGN1Ja8BHKcAIaiNAEBRioYBGWgQeTvhLEAEEnrgI2cQYhydln0AEfaZuUxSiDJ7ZB/zqtSi6r0npQ03TUIp0i0zBIGYIIvuA99vEZH/AVP/+sZTs1Bl7aNAtDLdOrgYuV7Me/KoVmC06/VWM9W1CDXcerRN51hxncQhzqW+dX/WwiE7HX8RbfyP8N9vCJbqC23sW7NMt0x/gRyAu9GDUo9rsdoyAfj8a2o5jeCif33VZ5YRXWYIP6sQNJOIIU5Jw+XEIXrjk/t8kluL1hKudxNcvwI7jDL6Ysku4=</latexit> x2 <latexit sha1_base64="Qqfga3YaBoUI0Yb7fQVzumPP1Sw=">AAAChHichVHLSsNAFD1GrbW+qm4EN8WiuJBy6xsXUnTjsg9rC1VKEqc1mCYhSQu1+AO6VVy4UnAhfoAf4MYfcNFPEJcKblx4mwZEi3rDZM6cuefOmbmKpWuOS9Tskrp7egN9wf7QwODQ8Eh4dGzHMau2KrKqqZt2XpEdoWuGyLqaq4u8ZQu5ougipxxutvZzNWE7mmlsu3VL7FXksqGVNFV2mUrVi+EoxciLSCeI+yAKP5Jm+B672IcJFVVUIGDAZaxDhsNfAXEQLOb20GDOZqR5+wLHCLG2ylmCM2RmD/lf5lXBZw1et2o6nlrlU3QeNisjmKYnuqVXeqQ7eqaPX2s1vBotL3WelbZWWMWRk4nM+7+qCs8uDr5Uf3p2UcKq51Vj75bHtG6htvW1o4vXzFp6ujFD1/TC/q+oSQ98A6P2pt6kRPryDz8Ke+EX4wbFf7ajE+zMx+LLMUotRhMbfquCmMQUZrkfK0hgC0lkub7AKc5wLgWkOWlBWmqnSl2+ZhzfQlr/BNcbj/U=</latexit> y sample max sample min This is why we need a criterion taking uncertainty into consideration instead of direct use of ML predicted values to guide exploration. It’ll be nice to gather more information around here even though the mean is not so high (the predictions have a large variance) <latexit sha1_base64="i+Yum6tkJlYfyXr5MYELEoyKzkM=">AAAChHichVFNLwNRFD3GV3222EhsGk3FQpo7rZZaSMPGUlGVIDIzHiadzkxmpk2q8QfYEgsrEgvxA/wAG3/Awk8QSxIbC3dmKmKBO3nzzjvvnvvOe1e1Dd31iJ7apPaOzq7uSE9vX//AYDQ2NLzuWjVHEyXNMixnQ1VcYeimKHm6Z4gN2xFKVTVEWa0s+vvlunBc3TLXvIYttqvKvqnv6ZriMVVs7MQSlErnc2nKx0OQzYVAzqfjcoqCSKAVy1bsDlvYhQUNNVQhYMJjbECBy98mZBBs5rbRZM5hpAf7AkfoZW2NswRnKMxW+L/Pq80Wa/Lar+kGao1PMXg4rIwjSY90Q6/0QLf0TB+/1moGNXwvDZ7VUCvsnejx6Or7v6oqzx4OvlV/evawh9nAq87e7YDxb6GF+vrh+evq3EqyOUFX9ML+L+mJ7vkGZv1Nuy6KlYs//KjshV+MG/TVhfjvYD2dknOpTHE6UVhotSqCMYxjkvsxgwKWsIwS1xc4wSnOpC5pSspI2TBVamtpRvAjpPlPiwOQTw==</latexit> y
44. / 35 27 #3. Optimism in the face of uncertainty <latexit sha1_base64="Nr2ImZk9Gvp+/1cBWPwAK6yswp0=">AAACiXichVFNLwNRFD0dX1VfxUZi02iIVXNHBOmq0Y1lP7QkiMyMh9H5ysy0UU3/gJWdYEViIX6AH2DjD1j0J4gliY2F2+kkguBO3rzzzrvnvvPeVR1D93yiVkTq6u7p7Yv2xwYGh4ZH4qNjZc+uupooabZhu+uq4glDt0TJ131DrDuuUEzVEGtqJdveX6sJ19Nta9WvO2LLVPYsfVfXFJ+p8qZqNg6b2/EkpSiIxE8ghyCJMHJ2/A6b2IENDVWYELDgMzagwONvAzIIDnNbaDDnMtKDfYEmYqytcpbgDIXZCv/3eLURshav2zW9QK3xKQYPl5UJTNMj3dALPdAtPdH7r7UaQY22lzrPakcrnO2R44ni278qk2cf+5+qPz372MVS4FVn707AtG+hdfS1o9OXYrow3ZihK3pm/5fUonu+gVV71a7zonDxhx+VvfCLcYPk7+34CcpzKXkhRfn5ZGY5bFUUk5jCLPdjERmsIIcS1z/ACc5wLg1IsrQkpTupUiTUjONLSNkPIVKSSQ==</latexit> x <latexit sha1_base64="g1U7Tzgl+Lj3TSOL2Mnl/fuxw90=">AAACi3ichVHLSsNAFL2Nr1qtrboR3BRLxVW5UVEpLooiuOzDPqAtJYljDc2LJC3W0B9w6cZF3Si4ED/AD3DjD7joJ4jLCm5ceJsGRIv1hsmcOXPPnTNzRUORLRux6+PGxicmp/zTgZnZ4FwoPL+Qt/SGKbGcpCu6WRQFiymyxnK2bCusaJhMUEWFFcT6fn+/0GSmJevakd0yWEUVapp8IkuCTVSxLKrOWbvKV8NRjKMbkWHAeyAKXqT08COU4Rh0kKABKjDQwCasgAAWfSXgAcEgrgIOcSYh2d1n0IYAaRuUxShDILZO/xqtSh6r0bpf03LVEp2i0DBJGYEYvuA99vAZH/AVP/+s5bg1+l5aNIsDLTOqoYul7Me/KpVmG06/VSM923ACO65XmbwbLtO/hTTQN8+vetlEJuas4i2+kf8b7OIT3UBrvkt3aZbpjPAjkhd6MWoQ/7sdwyC/Hue34pjejCb3vFb5YRlWYI36sQ1JOIQU5Nw+XEIHrrkgt8EluN1BKufzNIvwI7iDL6QMku0=</latexit> x1 <latexit sha1_base64="HrABz/ySZPgA1Ybj81k3XV8tH68=">AAACi3ichVHNSgJRFD5Of2aZVpugjSRGKzlaVEgLKYKW/uQPqMjMdLXB+WNmlGzwBVq2aWGbghbRA/QAbXqBFj5CtDRo06LjOBAl2Rnu3O9+93znfvceQZcl00LsebiJyanpGe+sb27evxAILi7lTa1piCwnarJmFAXeZLKkspwlWTIr6gbjFUFmBaFxMNgvtJhhSpp6bLV1VlH4uirVJJG3iCqWBcU+61Tj1WAYo+hEaBTEXBAGN1Ja8BHKcAIaiNAEBRioYBGWgQeTvhLEAEEnrgI2cQYhydln0AEfaZuUxSiDJ7ZB/zqtSi6r0npQ03TUIp0i0zBIGYIIvuA99vEZH/AVP/+sZTs1Bl7aNAtDLdOrgYuV7Me/KoVmC06/VWM9W1CDXcerRN51hxncQhzqW+dX/WwiE7HX8RbfyP8N9vCJbqC23sW7NMt0x/gRyAu9GDUo9rsdoyAfj8a2o5jeCif33VZ5YRXWYIP6sQNJOIIU5Jw+XEIXrjk/t8kluL1hKudxNcvwI7jDL6Ysku4=</latexit> x2 <latexit sha1_base64="Qqfga3YaBoUI0Yb7fQVzumPP1Sw=">AAAChHichVHLSsNAFD1GrbW+qm4EN8WiuJBy6xsXUnTjsg9rC1VKEqc1mCYhSQu1+AO6VVy4UnAhfoAf4MYfcNFPEJcKblx4mwZEi3rDZM6cuefOmbmKpWuOS9Tskrp7egN9wf7QwODQ8Eh4dGzHMau2KrKqqZt2XpEdoWuGyLqaq4u8ZQu5ougipxxutvZzNWE7mmlsu3VL7FXksqGVNFV2mUrVi+EoxciLSCeI+yAKP5Jm+B672IcJFVVUIGDAZaxDhsNfAXEQLOb20GDOZqR5+wLHCLG2ylmCM2RmD/lf5lXBZw1et2o6nlrlU3QeNisjmKYnuqVXeqQ7eqaPX2s1vBotL3WelbZWWMWRk4nM+7+qCs8uDr5Uf3p2UcKq51Vj75bHtG6htvW1o4vXzFp6ujFD1/TC/q+oSQ98A6P2pt6kRPryDz8Ke+EX4wbFf7ajE+zMx+LLMUotRhMbfquCmMQUZrkfK0hgC0lkub7AKc5wLgWkOWlBWmqnSl2+ZhzfQlr/BNcbj/U=</latexit> y sample max sample min This is why we need a criterion taking uncertainty into consideration instead of direct use of ML predicted values to guide exploration. <latexit sha1_base64="Qqfga3YaBoUI0Yb7fQVzumPP1Sw=">AAAChHichVHLSsNAFD1GrbW+qm4EN8WiuJBy6xsXUnTjsg9rC1VKEqc1mCYhSQu1+AO6VVy4UnAhfoAf4MYfcNFPEJcKblx4mwZEi3rDZM6cuefOmbmKpWuOS9Tskrp7egN9wf7QwODQ8Eh4dGzHMau2KrKqqZt2XpEdoWuGyLqaq4u8ZQu5ougipxxutvZzNWE7mmlsu3VL7FXksqGVNFV2mUrVi+EoxciLSCeI+yAKP5Jm+B672IcJFVVUIGDAZaxDhsNfAXEQLOb20GDOZqR5+wLHCLG2ylmCM2RmD/lf5lXBZw1et2o6nlrlU3QeNisjmKYnuqVXeqQ7eqaPX2s1vBotL3WelbZWWMWRk4nM+7+qCs8uDr5Uf3p2UcKq51Vj75bHtG6htvW1o4vXzFp6ujFD1/TC/q+oSQ98A6P2pt6kRPryDz8Ke+EX4wbFf7ajE+zMx+LLMUotRhMbfquCmMQUZrkfK0hgC0lkub7AKc5wLgWkOWlBWmqnSl2+ZhzfQlr/BNcbj/U=</latexit> y <latexit sha1_base64="Qqfga3YaBoUI0Yb7fQVzumPP1Sw=">AAAChHichVHLSsNAFD1GrbW+qm4EN8WiuJBy6xsXUnTjsg9rC1VKEqc1mCYhSQu1+AO6VVy4UnAhfoAf4MYfcNFPEJcKblx4mwZEi3rDZM6cuefOmbmKpWuOS9Tskrp7egN9wf7QwODQ8Eh4dGzHMau2KrKqqZt2XpEdoWuGyLqaq4u8ZQu5ougipxxutvZzNWE7mmlsu3VL7FXksqGVNFV2mUrVi+EoxciLSCeI+yAKP5Jm+B672IcJFVVUIGDAZaxDhsNfAXEQLOb20GDOZqR5+wLHCLG2ylmCM2RmD/lf5lXBZw1et2o6nlrlU3QeNisjmKYnuqVXeqQ7eqaPX2s1vBotL3WelbZWWMWRk4nM+7+qCs8uDr5Uf3p2UcKq51Vj75bHtG6htvW1o4vXzFp6ujFD1/TC/q+oSQ98A6P2pt6kRPryDz8Ke+EX4wbFf7ajE+zMx+LLMUotRhMbfquCmMQUZrkfK0hgC0lkub7AKc5wLgWkOWlBWmqnSl2+ZhzfQlr/BNcbj/U=</latexit> y <latexit sha1_base64="uHwbcp1I7cfAojOCGzVZ2+fOJ7M=">AAACh3ichVE7TsNAEH2Yb/glQINEExGBoAnjACGh4tNQQiCAFKLINhuwcGzLdiKFiAtQ0AaJCiQKxAE4AA0XoMgRECVINBSMnSBEEZjV7s6+mTf7dke1Dd31iBodUmdXd09vX6h/YHBoOBwZGd11rbKjiaxmGZazryquMHRTZD3dM8S+7QilpBpiTz1Z9+N7FeG4umXueFVb5EvKkakXdU3xfMieqc4WIjGKp1NJWkhEKU6USlCSnUWS03I6KjPiWwwt27QiDzjAISxoKKMEARMe+wYUuDxykEGwGcujxpjDnh7EBc7Qz9wyZwnOUBg94fWIT7kWavLZr+kGbI1vMXg6zIxiip7pjt7oie7phT7b1qoFNXwtVd7VJlfYhfD5+PbHv6wS7x6Of1h/avZQRCrQqrN2O0D8V2hNfuW0/ra9nJmqTdMNvbL+a2rQI7/ArLxrt1sic/WHHpW18I9xg767EG3v7CbicjI+v7UQW1lrtaoPE5jEDPdjCSvYwCayXP8YF6jjUgpJc1JSSjVTpY4WZwy/TFr9AnUXkR0=</latexit> p(y) <latexit sha1_base64="xAYAr5eatCRkt85fJM86k8bLQd0=">AAAChnichVE7TsNAEH2YXwi/AA0STUQEQhTWOCQhoYqgoSSEABI/2WYTLBzbsp1IIeICSLRQUIFEgTgAB6DhAhQcAVGCREPB2AlCFMCsdnf2zbzZtzuaYxqeT/TUIXV2dff0Rvqi/QODQ8OxkdF1z665uijptmm7m5rqCdOwRMk3fFNsOq5Qq5opNrTDpSC+UReuZ9jWmt9wxE5VrVhG2dBVn6FiY3d2L5YgOZfNUCoZJ5kom6QMO2lSckourjASWAJtW7Fjd9jGPmzoqKEKAQs++yZUeDy2oIDgMLaDJmMue0YYFzhGlLk1zhKcoTJ6yGuFT1tt1OJzUNML2TrfYvJ0mRnHFD3SDb3SA93SM338WqsZ1gi0NHjXWlzh7A2fjBff/2VVefdx8M36U7OPMrKhVoO1OyESvEJv8etH56/FhdWp5jRd0Qvrv6QnuucXWPU3/bogVi/+0KOxFv4xbtBXF+K/O+tJWcnIc4VUIr/YblUEE5jEDPdjHnksYwUlrl/BKc5wLkUkWUpL861UqaPNGcMPk/Kf1ZuQ2g==</latexit> y⇤ <latexit sha1_base64="Cx8vtQI6wD6rFemnJtzP0VxR74M=">AAACp3ichVE9T9tQFD0Y2vJRSqALUheLiCphiK5TGhIkKlSWboSPhEh8RLbzQp9wbMt+ieRG7FX/QIdOrdQBIXWFnYU/wMBPqDqCxMLQaydVxQBc6/mdd9499533ruU7MlRElwPa4NCTp8+GR0bHno+/mEhNTlVDrx3YomJ7jhfULDMUjnRFRUnliJofCLNlOWLLOliJ97c6Igil526qyBe7LXPflU1pm4qpemqmnIneRXtz2aUd6ap6l+HhHsOminQ/E2X1RlRPpSlXKhZoPq9TjqiYpwKDt2SUjJJuMBNHGv0oe6kT7KABDzbaaEHAhWLswETI3zYMEHzmdtFlLmAkk32BQ4yyts1ZgjNMZg/4v8+r7T7r8jquGSZqm09xeASs1DFLF3REV3ROx/Sbbu+t1U1qxF4inq2eVvj1iS/TGzePqlo8K3z8r3rQs0ITxcSrZO9+wsS3sHv6zqevVxuL67Pd1/SD/rD/73RJZ3wDt3Nt/1wT698e8GOxF34xbtC/Luj3g2o+ZxRyb9bm08vv+60axivMIMP9WMAyPqCMCtf/jF84wamW1Va1qlbrpWoDfc1L3AnN/AuqB50Q</latexit> P(y > y⇤) = Z 1 y⇤ p(y)dy <latexit sha1_base64="svoh2wRjAJ9IgkLWLNY4QZ6NPX0=">AAACwnichVHLTttAFL24vMsjlA0Sm1EjKlpF0XWANEECIWilLnk0gBSHyHYmMOBX7UkkY/ID/AALVi3qouoH9AO6abetWPAJqEsqsWHRayeoYgFcazxnztxz58xcw7NEIBEvupQn3T29ff0Dg0+HhkdGU2PPNgO34Zu8ZLqW628besAt4fCSFNLi257Pdduw+JZxsBLvbzW5HwjXeS9Dj1dsfdcRdWHqkqhqakGzdblnGNHbVlnLhEzLHGkZFi6GO68qC5pwZDUi2NohWJchowSz5krmTYcvWS2sptKYLRbyOJtjmEUs5DBPYA7VolpkKjFxpKETq27qG2hQAxdMaIANHByQhC3QIaCvDCogeMRVICLOJySSfQ4tGCRtg7I4ZejEHtB/l1blDuvQOq4ZJGqTTrFo+KRkMIXn+AWv8Ad+xUu8ubdWlNSIvYQ0G20t96qjxxMb14+qbJol7P1XPehZQh0KiVdB3r2EiW9htvXNw5Orjfn1qegFfsI/5P8jXuB3uoHT/Gt+XuPrpw/4McgLvRg16LYL7H6wmcuq+ezM2mx6abnTqn6YhOcwTf14DUvwDlahRPXP4Cf8gt/KG2Vf+aAE7VSlq6MZhzuhHP0DArmnyA==</latexit> E[ y | y > y⇤] = Z 1 y⇤ y · p(y)dy Probability of improvement (PI) Expected Improvement (EI) Upper Confidence Bound (UCB) Optimization objectives under uncertainty (aka acquisition function) It’ll be nice to gather more information around here even though the mean is not so high (the predictions have a large variance) <latexit sha1_base64="i+Yum6tkJlYfyXr5MYELEoyKzkM=">AAAChHichVFNLwNRFD3GV3222EhsGk3FQpo7rZZaSMPGUlGVIDIzHiadzkxmpk2q8QfYEgsrEgvxA/wAG3/Awk8QSxIbC3dmKmKBO3nzzjvvnvvOe1e1Dd31iJ7apPaOzq7uSE9vX//AYDQ2NLzuWjVHEyXNMixnQ1VcYeimKHm6Z4gN2xFKVTVEWa0s+vvlunBc3TLXvIYttqvKvqnv6ZriMVVs7MQSlErnc2nKx0OQzYVAzqfjcoqCSKAVy1bsDlvYhQUNNVQhYMJjbECBy98mZBBs5rbRZM5hpAf7AkfoZW2NswRnKMxW+L/Pq80Wa/Lar+kGao1PMXg4rIwjSY90Q6/0QLf0TB+/1moGNXwvDZ7VUCvsnejx6Or7v6oqzx4OvlV/evawh9nAq87e7YDxb6GF+vrh+evq3EqyOUFX9ML+L+mJ7vkGZv1Nuy6KlYs//KjshV+MG/TVhfjvYD2dknOpTHE6UVhotSqCMYxjkvsxgwKWsIwS1xc4wSnOpC5pSspI2TBVamtpRvAjpPlPiwOQTw==</latexit> y
45. / 35 28 Identifying local peaks of EI of the ML model Expected improvement Local peaks of EI would be nice candidates having locally maximal EIs. But they are not at given sample points, and the following local search is designed. multistart from given sample points adding small random perturbation, and update position when EI increases stop when local perturbation doesn’t change the EI value any more. run clustering over final candidates, and suggest K candidates having locally maximal EI values. Every time SWED is changed, the corresponding composition is estimated by our algorithm, and then recalculate valid SWED from it. Partly because tree ensemble regression functions are locally bumpy, this clustering is effective <latexit sha1_base64="Nr2ImZk9Gvp+/1cBWPwAK6yswp0=">AAACiXichVFNLwNRFD0dX1VfxUZi02iIVXNHBOmq0Y1lP7QkiMyMh9H5ysy0UU3/gJWdYEViIX6AH2DjD1j0J4gliY2F2+kkguBO3rzzzrvnvvPeVR1D93yiVkTq6u7p7Yv2xwYGh4ZH4qNjZc+uupooabZhu+uq4glDt0TJ131DrDuuUEzVEGtqJdveX6sJ19Nta9WvO2LLVPYsfVfXFJ+p8qZqNg6b2/EkpSiIxE8ghyCJMHJ2/A6b2IENDVWYELDgMzagwONvAzIIDnNbaDDnMtKDfYEmYqytcpbgDIXZCv/3eLURshav2zW9QK3xKQYPl5UJTNMj3dALPdAtPdH7r7UaQY22lzrPakcrnO2R44ni278qk2cf+5+qPz372MVS4FVn707AtG+hdfS1o9OXYrow3ZihK3pm/5fUonu+gVV71a7zonDxhx+VvfCLcYPk7+34CcpzKXkhRfn5ZGY5bFUUk5jCLPdjERmsIIcS1z/ACc5wLg1IsrQkpTupUiTUjONLSNkPIVKSSQ==</latexit> x
46. / 35 29 Explorative Search with SWED SWED represent every element with respect to a given set of elemental descriptors. So we can focus only on the selected elemental properties to explore catalysts. 74 elements Compositional (onehot-like) Catalyst: Mg 83.46, Li 16.53 SWED-8 83.46 × 16.53 × 0.00 0.00 … SWED-3 83.46 × 16.53 × 0.00 0.00 … SWED-3 features: electronegativity, density, enthalpy of fusion SWED-8 features: SWED-3 features + atomic weight, atomic radius, m.p., b.p., ionization enegy Each user’s intention and focus for catalyst exploration can be design through the elemental descriptor choice. can control specificity & focus
47. / 35 30 Our updated dataset The original dataset: 1866 catalyst records from 421 reports (1982 - 2009) Mine, S.; Takao, M.; Yamaguchi, T.; Toyao, T.*; Maeno, Z.; Hakim Siddiki, S. M. A.; Takakusagi, S.; Shimizu, K.*; Takigawa, I.* ChemCatChem 2021. https://doi.org/10.1002/cctc.202100495. 4559 catalyst records from 542 reports The update dataset: 4559 catalyst records from 542 reports (2010 - 2019)
48. / 35 31 ML Predictions of C2 yields 1. Conventional: composition + condition 2. Proposed(Exploitative): composition + SWED + condition 3. Proposed(Explorative): SWED + condition w/ SWED→composition estimator RFR (Random Forest); ETR (ExtraTrees); XGB (XGBoost) SWED-3 features: electronegativity, density, enthalpy of fusion SWED-8 features: SWED-3 features + atomic weight, atomic radius, m.p., b.p., ionization enegy
49. / 35 32 Top 20 highest-EI candidates based on SWED-3 As appeared not included in the data Fs, Se, Os, Bm infrequent elements also observed though these are toxic and impractical but explorative suggestions were able be made
50. / 35 33 Post analysis for models and suggested catalysts 1st: (1) Mn: 72.3 (2) Li: 27.7 2nd: (1) Sr:50.0 (2) Ce:45.0 (3) Yb:5.0 With SHAP, feature importance/permutation importance, dependency plot, etc.
51. / 35 34 Today’s talk Gas-phase reactions on sold-phase catalyst surface (Heterogeneous catalysis) Industrial Synthesis (e.g. Haber-Bosch), Automobile Exhaust Gas Purification, Methane Conversion, etc. Reactants (Gas) Catalysts (Solid) Nano-particle surface High Temperature, High Pressure Adsorption Diffusion Dissociation Recombination Desorption God made the bulk; the surface was invented by the devil Devilishly complex too-many-factor process!! —— Wolfgang Pauli Our struggles for better ML practices with underspecified, sparse, biased observational data (i.e. a collection of experimental facts from literature)
52. / 35 35 Acknowledgements Ken-ichi SHIMIZU Satoru TAKAKUSAGI Takashi TOYAO Zen MAENO Keisuke SUZUKI Motoshi TAKAO Shinya MINE Taichi YAMAGUCHI S. M. A. Hakim Siddiki