決定森回帰の信頼区間推定, Benign Overfitting, 多変量木とReLUネットの入力空間分割

寸㹀啾㔐䌓ך⥋걾⼒꟦䱿㹀 #FOJHO0WFSUUJOH
㢳㢌ꆀ加ה3F-6طحزךⰅ⸂瑞꟦ⴓⶴ
+45$3&45㷕统侧椚ٌرٕח㛇בֻ儗瑞꟦㾜Ꟛ㘗،٦ؗذؙثٍךⶼ⳿ה䘔欽劤募$3&45

堣唒㷕统棳ؿٖؓأزٙ٦ؙءّحف鑧겗䲿⣘
椚⻉㷕灇瑔䨽ꬠ倜濼腉窟さ灇瑔إٝة٦!❨ꢻ㣽"53 J14稢脄鸬䵿棳

⻌嵲麣㣐㷕⻉㷕⿾䘔ⶼ䧭灇瑔䬿挿 *$3F%%

戣䊛♧㷕
[email protected]
2022䎃2剢24傈

View Slide

猘ךꟼ䗰ꨄ侔圓鸡׾⠵ֲ堣唒㷕统ر٦ة⚥䗰㘗荈搫猰㷕
ꨄ侔圓鸡穈さׇ涸圓鸡װ➿侧涸圓鸡

꧊さծ锷椚ծ纇ծ갫⴨٥穈さׇծ禸⴨俑㶵⴨ծخٔ٦ծؚٓؿծ穈さׇ䎗⡦ծ˘
㼎韋חꨄ侔圓鸡㼎韋꟦חꨄ侔圓鸡
堣唒㷕统ٌرٕחꨄ侔圓鸡劤傈ך鑧겗
㹋⹡כ䎌稢脄歗⫷ך帾㾴㷕统椚灇
ה⻉㷕⻌㣐

View Slide

➙傈ך鑧겗䲿⣘
˖ 寸㹀啾㔐䌓ך⥋걾⼒꟦䱿㹀٥#FOJHO0WFSUUJOH
噟⹡ 荈搫猰㷕דך堣唒㷕统ⵃ崞欽
דِ٦ؠה׃ג寸㹀加،ٝ؟ٝـٕ
הصُ٦ٕٓطحز
׾⢪׏גְג⳿⠓׏׋植韋ה㉏겗ך稱➜
˖ 㢳㢌ꆀ加ה3F-6طحزךⰅ⸂瑞꟦ⴓⶴ

View Slide

خٔ٦׏ג葺ְ״י˘
寸㹀加ծ禸窟埠ծر٦ة圓鸡ծؿ؋؎ٕءأذيծ9.-ծ

View Slide

ꨄ侔圓鸡ד㹀纏ׁ׸׷堣唒㷕统ٌرٕ
寸㹀啾
寸㹀加،ٝ؟ٝـٕ
صُ٦ٕٓطحزٙ٦ؙ
锷椚䱿锷٥ٕ٦ٕك٦أ٥䩛竲ֹ㘗זוך 傊⚅➿㘗ך
堣唒㷕统ٌرٕ
˖ 輐さ٥涪㾜ָ➂䊨濼腉ⴓꅿך剑㣐ךꟼ䗰✲
˖ ؔ٦زوزٝ٥锷椚㔐騟ה娖〷涸חכずׄ⳿涪挿
娖〷涸ח㺘䱸זꟼ⤘

View Slide

寸㹀加ה锷椚ꟼ侧
if x2 ≤ θ1 then
if x1 ≤ θ2 then
return Blue
else
if x2 ≤ θ4 then
return Red
else
if x1 ≤ θ5 then
return Red
else
if x1 ≤ θ6 then
return Blue
else
return Red
else
if x1 ≤ θ3 then
if x2 ≤ θ7 then
return Blue
else
return Blue
else
return Red
AAAClnichVHLSsNAFD3Gd3201Y3gplgUV2UiRcWFFEV02YfVgpWQxLENpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxGUFNy68TQOiot4wmTNn7rlzZq7mmIYnGGt0SJ1d3T29ff2hgcGh4XAkOrLh2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvanvLrf3NGnc9w7bWxaHDtytqyTJ2DV0VRCmRaEGZiRVNvh8rijIXqiIrkThLMD9iP4EcgDiCSNuRexSxAxs6qqiAw4IgbEKFR98WZDA4xG2jTpxLyPD3OY4RIm2VsjhlqMTu0b9Eq62AtWjdqun5ap1OMWm4pIxhkj2xW9Zkj+yOPbP3X2vV/RotL4c0a20td5TwyVju7V9VhWaB8qfqT88Cu5j3vRrk3fGZ1i30tr52dNrMLWQn61Pskr2Q/wvWYA90A6v2ql9lePbsDz8aeaEXowbJ39vxE2zMJOTZRDKTjKeWglb1YRwTmKZ+zCGFNaSRp/oHOMc1bqQxaVFakVbbqVJHoBnFl5DSHzOslho=
X2
 ✓1
yes no
AAAClnichVHLSsNAFD3Gd3201Y3gplgUV2UiRcWFFEV02YfVgpWQxLENpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxGUFNy68TQOiot4wmTNn7rlzZq7mmIYnGGt0SJ1d3T29ff2hgcGh4XAkOrLh2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvanvLrf3NGnc9w7bWxaHDtytqyTJ2DV0VRCmRaEGRY0WT78eKosyFqswokThLMD9iP4EcgDiCSNuRexSxAxs6qqiAw4IgbEKFR98WZDA4xG2jTpxLyPD3OY4RIm2VsjhlqMTu0b9Eq62AtWjdqun5ap1OMWm4pIxhkj2xW9Zkj+yOPbP3X2vV/RotL4c0a20td5TwyVju7V9VhWaB8qfqT88Cu5j3vRrk3fGZ1i30tr52dNrMLWQn61Pskr2Q/wvWYA90A6v2ql9lePbsDz8aeaEXowbJ39vxE2zMJOTZRDKTjKeWglb1YRwTmKZ+zCGFNaSRp/oHOMc1bqQxaVFakVbbqVJHoBnFl5DSHzOelho=
X1
 ✓2
yes no
Blue
AAAClnichVHLSsNAFD2N7/qquim4KZaKqzItRcWFiCK6rNZqoZWQxGkbTJOYTCta/AF/wIW4UFARP8APcOMPuOgniMsKblx4mwZExXrDZM6cuefOmbmqbeiuYKwRkLq6e3r7+geCg0PDI6OhsfFt16o6Gs9qlmE5OVVxuaGbPCt0YfCc7XClohp8R91fae3v1Ljj6pa5JY5svltRSqZe1DVFECWHxnJyMlIw+EGkIMpcKHJKDkVZnHkR+Q0SPojCj7QVekABe7CgoYoKOEwIwgYUuPTlkQCDTdwu6sQ5hHRvn+MEQdJWKYtThkLsPv1LtMr7rEnrVk3XU2t0ikHDIWUEMfbM7liTPbF79sI+/qxV92q0vBzRrLa13JZHT8OZ939VFZoFyl+qjp4Fipj3vOrk3faY1i20tr52fNbMLGzG6tPsir2S/0vWYI90A7P2pl1v8M3zDn5U8kIvRg1K/GzHb7CdjCdm46mNVHRp2W9VPyYxhRnqxxyWsI40slT/EBe4wa0UlhalVWmtnSoFfM0EvoWU/gQ6DJYd
X2
 ✓4
yes no
Red AAAClnichVHLSsNAFD3GV62vVjeCm2JRXJWJ+MKFFEV0WVurBSshiVMNTZOYTCta/AF/wIW4UFARP8APcOMPuOgniEsFNy68SQOiot4wmTNn7rlzZq7mmIYnGGu0SK1t7R2dka5od09vX38sPrDu2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvaOVFf3+jxl3PsK01ceDwrYq6YxklQ1cFUUosXlDkRNHke4mi2OVCVaaUWJKlWBCJn0AOQRJhZOzYHYrYhg0dVVTAYUEQNqHCo28TMhgc4rZQJ84lZAT7HEeIkrZKWZwyVGLL9N+h1WbIWrT2a3qBWqdTTBouKRMYZY/shr2wB3bLntj7r7XqQQ3fywHNWlPLHaX/eCj39q+qQrPA7qfqT88CJcwGXg3y7gSMfwu9qa8dnrzk5rKj9TF2wZ7J/zlrsHu6gVV71S9Xefb0Dz8aeaEXowbJ39vxE6xPpOTp1OTqZDK9ELYqgmGMYJz6MYM0VpBBnurv4wxXuJaGpHlpSVpupkotoWYQX0LKfAA5/pYd
X1
 ✓5
yes no
Red AAAClnichVHLSsNAFD3Gd321uhHcFIviqkykVHEhRRFd9mG1YCUkcWyDaRKTaUWLP+APuBAXCiriB/gBbvwBF/0EcVnBjQtv04CoqDdM5syZe+6cmas5puEJxhodUmdXd09vX39oYHBoeCQcGd3w7Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tb3l1v5mjbueYVvr4tDh2xW1ZBm7hq4KopRwpKDI0aLJ96NFUeZCVZJKOMbizI/oTyAHIIYg0nb4HkXswIaOKirgsCAIm1Dh0bcFGQwOcduoE+cSMvx9jmOESFulLE4ZKrF79C/RaitgLVq3anq+WqdTTBouKaOYYk/sljXZI7tjz+z911p1v0bLyyHNWlvLHWXkZDz39q+qQrNA+VP1p2eBXcz7Xg3y7vhM6xZ6W187Om3mFrJT9Wl2yV7I/wVrsAe6gVV71a8yPHv2hx+NvNCLUYPk7+34CTZm43IynsgkYqmloFV9mMAkZqgfc0hhDWnkqf4BznGNG2lcWpRWpNV2qtQRaMbwJaT0Bzwelh4=
X1
 ✓6
yes no
Red
Blue
X1
 ✓3
yes no
Red
X1
 ✓7
yes no
Blue Blue
AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmNagEi6AIlnmYByQh7K5jMmRf7E4CMfgDljYWsVGwED/AD7DxByzyCWIZwcbCu5sF0WC8y+ycOXPPnTNzZVNlNiek7xMmJqemZ/yzgbn54EIovLhUsI2WpdC8YqiGVZIlm6pMp3nOuEpLpkUlTVZpUW4eOPvFNrVsZujHvGPSqibVdXbKFIkjVarwBuVSTayFoyRO3IiMAtEDUfAibYQfoQInYIACLdCAgg4csQoS2PiVQQQCJnJV6CJnIWLuPoVzCKC2hVkUMyRkm/iv46rssTqunZq2q1bwFBWHhcoIxMgLuScD8kweyCv5/LNW163heOngLA+11KyFLlZyH/+qNJw5NL5VYz1zOIVd1ytD76bLOLdQhvr22dUgl8zGuuvklryh/xvSJ094A739rtxlaLY3xo+MXvDFsEHi73aMgsJmXNyOJzKJaGrfa5UfVmENNrAfO5CCI0hD3u3DJfTgWggKW0JS2BumCj5Psww/Qjj8Ak3rksg=
✓1
AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmNagEi6AIlnmYByQh7K5jMmRf7E4CMfgDljYWsVGwED/AD7DxByzyCWIZwcbCu5sF0WC8y+ycOXPPnTNzZVNlNiek7xMmJqemZ/yzgbn54EIovLhUsI2WpdC8YqiGVZIlm6pMp3nOuEpLpkUlTVZpUW4eOPvFNrVsZujHvGPSqibVdXbKFIkjVarwBuVSLVELR0mcuBEZBaIHouBF2gg/QgVOwAAFWqABBR04YhUksPErgwgETOSq0EXOQsTcfQrnEEBtC7MoZkjINvFfx1XZY3VcOzVtV63gKSoOC5URiJEXck8G5Jk8kFfy+WetrlvD8dLBWR5qqVkLXazkPv5VaThzaHyrxnrmcAq7rleG3k2XcW6hDPXts6tBLpmNddfJLXlD/zekT57wBnr7XbnL0GxvjB8ZveCLYYPE3+0YBYXNuLgdT2QS0dS+1yo/rMIabGA/diAFR5CGvNuHS+jBtRAUtoSksDdMFXyeZhl+hHD4BVRLkss=
✓4
AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRAooZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYi9eCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4V4NLYTTWQS4dSB2yovrMMGbGE/diEFx5CGvNOHS+jDteAXtoWksD9KFTyuZhV+hHD0BVALksk=
✓2
AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRQfIRYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixPS9wgTk1PTM95Z39y8fyEQXFzKW3rLlGlO1hXdLEqiRRWm0RxnXKFFw6SiKim0IDUP7f1Cm5oW07UT3jFoRRXrGqsxWeRIFcu8QblY3a4GwyRKnAiNgpgLwuBGSg8+QhlOQQcZWqACBQ04YgVEsPArQQwIGMhVoIuciYg5+xTOwYfaFmZRzBCRbeK/jquSy2q4tmtajlrGUxQcJipDECEv5J4MyDN5IK/k889aXaeG7aWDszTUUqMauFjJfvyrUnHm0PhWjfXMoQZ7jleG3g2HsW8hD/Xts6tBNpGJdNfJLXlD/zekT57wBlr7Xb5L00xvjB8JveCLYYNiv9sxCvKb0dhONJ6Oh5MHbqu8sAprsIH92IUkHEMKck4fLqEH14Jf2BISwv4wVfC4mmX4EcLRF1Zrksw=
✓5
AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRQgoZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYS9SCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4XtaCwRjWfi4dSB2yovrMMGbGE/diEFx5CGvNOHS+jDteAXdoSksD9KFTyuZhV+hHD0BViLks0=
✓6
AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmQQooZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYi9eCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4XtaGwnmsgkwqkDt1VeWIcN2MJ+7EIKjiENeacPl9CHa8EvxIWksD9KFTyuZhV+hHD0BVIrkso=
✓3
AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRQIoZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYS9SCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4XtaGw3Gs/Ew6kDt1VeWIcN2MJ+JCAFx5CGvNOHS+jDteAXdoSksD9KFTyuZhV+hHD0BVqrks4=
✓7
AAACsnichVFNLwNBGH6sr/ouLhKXjYa4aGZpEIlEuDi2KEJls7sdNbHd3exOK9X4A64ODk4kDuIHiCsXf8DBTxBHEhcHb7crguDdzM4zz7zPO8/Ma3q2CCRjDw1KY1NzS2usrb2js6u7J97btxq4Jd/iWcu1XX/dNAJuC4dnpZA2X/d8bhRNm6+Zuwu1/bUy9wPhOiuy4vGtolFwxLawDEmUHh9L60KdmVUzuqbm9ni+wAmOf8KJD5jLuzLQ4wmWZGGoP4EWgQSiSLvxK+SQhwsLJRTB4UAStmEgoG8TGhg84rZQJc4nJMJ9jgO0k7ZEWZwyDGJ36V+g1WbEOrSu1QxCtUWn2DR8UqoYZvfsgj2zO3bJHtnbr7WqYY2alwrNZl3LPb3ncGD59V9VkWaJnU/Vn54ltjEdehXk3QuZ2i2sur68f/y8PLM0XB1hZ+yJ/J+yB3ZLN3DKL9Z5hi+d/OHHJC/0YtQg7Xs7foLV8aQ2mUxlUom5+ahVMQxiCKPUjynMYRFpZKn+Ea5xg1slpWwohmLVU5WGSNOPL6HY7+R7n+w=
Pi := Q1
^ Q2
^ Q3
^ . . .
AAACynichVHLTttAFD24DyB9kJYNUjdWIxCraIIQIKRWUatKLFgQIBAJI8s2lzBkYrv2JFJqZceqP9BFV0ViUfEB/YBuumXBgvIFiCWVuumCG8cSAgS91njOnLnnzpm5bqhkrIU4GTAePHz0eHBoOPfk6bPnI/kXL9fioBV5VPUCFUQ114lJSZ+qWmpFtTAip+kqWncb73v7622KYhn4q7oT0mbTqftyW3qOZsrOf6jYu+b8G9NyqS79xONScTdXsxumpeijaekd0o6tTMtKybdXTM4ifysT2PmCKIo0zNuglIECslgK8j9gYQsBPLTQBMGHZqzgIOZvAyUIhMxtImEuYiTTfUIXOda2OIs4w2G2wf86rzYy1ud1r2acqj0+RfGIWGliXByL7+JC/BKH4kz8u7NWktboeenw7Pa1FNojn8dW/v5X1eRZY+dKda9njW3MpV4lew9TpncLr69vf/pysTK/PJ5MiH1xzv6/iRPxk2/gt/94BxVa/nqPH5e98Itxg0o323EbrE0VSzPF6cp0ofwua9UQXuE1JrkfsyhjAUuocv1DHOE3To1FIzI6RtJPNQYyzSiuhbF3CZqVqlw=
Qj :=
(
Xk
 ✓l
Xk > ✓l
寸㹀加כ锷椚䒭ה׃ג鼅鎉垥彊䕎琎ㄤ䕎

Blue
AAACoXichVHLSsNAFD3G97NVN4IugqXiqkyKqAiC6EZ3tbZaaEtJ0rGGpklIpgUtblz6Ay5cKQiKK934AW78ARd+gris4MaFt2lAtFhvmMyZM/fcOTNXc0zDE4y9dEndPb19/QODQ8Mjo2Oh8PjErmdXXZ2nddu03Yymetw0LJ4WhjB5xnG5WtFMvqeVN5r7ezXueoZtpcShw/MVtWQZ+4auCqIK4ZmUvLIqJwqKnKtxTiDeArmiLbxCOMJizA+5HSgBiCCIhB1+QA5F2NBRRQUcFgRhEyo8+rJQwOAQl0edOJeQ4e9zHGOItFXK4pShElumf4lW2YC1aN2s6flqnU4xabiklBFlz+yGNdgTu2Wv7PPPWnW/RtPLIc1aS8udQuh0aufjX1WFZoGDb1VHzwL7WPa9GuTd8ZnmLfSWvnZ01thZSUbrc+ySvZH/C/bCHukGVu1dv9rmyfMOfjTyQi9GDVJ+t6Md7MZjymJsYXshsrYetGoA05jFPPVjCWvYRAJpqn+Ca9zhXopIW1JCSrZSpa5AM4kfIWW/AAtRmVk=
T := P1
_ P2
_ . . .
AAACqXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuoxCSlpmfmVScDzSiu5QqxNYyJUQBSBlwxqXkpUOF4AWUDPQMwUMBkGEIZygxQEJAvsJ0hhiGFIZ8hmaGUIZchlSGPoQTIzmFIZCgGwmgGQwYDhgKgWCxDNVCsCMjKBMunMtQycAH1lgJVpQJVJAJFs4FkOpAXDRXNA/JBZhaDdScDbckB4iKgTgUGVYOrBisNPhucMFht8NLgD06zqsFmgNxSCaSTIHpTC+L5uySCvxPUlQukSxgyELrwurmEIY3BAuzWTKDbC8AiIF8kQ/SXVU3/HGwVpFqtZrDI4DXQ/QsNbhocBvogr+xL8tLA1KDZeNyTBHQLMMSAEWSIHh2YjDAjPUMzPZNAE2UHJ2hUcTBIMygxaADjw5zBgcGDIYAhFGh+F8MGhp0Mu5i0mQKZIpiiIEqZGKF6hBlQAFMyAOxdnYo=
(
T = 1
T = 0
Path
Query
Not Blue (= Red)

View Slide

跐駈锷椚ה؝ٝؾُ٦ةה鎘皾堣猰㷕ה牞穗㔐騟ה➂䊨濼腉
"'4"❨鿪⠓陽ד㿊劤畍⽆⯓欰׋׍ח侄ִג׮׵׏׋
כׄ׭ג➂䊨濼腉הְֲֿהלָ㹀
纏ׁ׸׋ֿהחז׏גְ׷չت٦زو
أ⠓陽պתדך鎘皾堣嚊䙀ך娖〷
˖ չ➂꟦ך״ֲח䙼罋דֹ׷堣唒պ׾湡䭷
׃ג植㖈ךչ鎘皾堣պ嚊䙀ח荚׷תד
˖ صُ٦ٕٓطحزٙ٦ؙծؔ٦زوزٝծ
䕎䒭鎉铂ծ鎘皾椚锷ծ锷椚㔐騟ծזוכ
ׅץגךずׄ⳿涪挿׾䭯א
˖ غ٦ؙٔ٦ծؐ؍٦ش٦ծؿؓٝ٥ظ؎
وٝծثُ٦ؚٔٝծؙٔ٦طծءٍظ
ٝծوؕٗحؙ٥ؾحخծ׋׍ך娖〷

View Slide

ֿךאכ简䕎ٌرٕח如ּ植➿ך䘔欽ر٦ة猰㷕ך⚺麣Ⱗ
https://www.kaggle.com/kaggle-survey-2021
Ԩ 넝礵䏝
Ԩ 넝鸞
Ԩ ꬊ简䕎
Ԩ ؕذ؞ٕٔؕ㢌侧ך䪔ְ
Ԩ ءٝفٕ٥鍑ꅸ׃װְׅ
Ԩ ⚛⴨⻉׃װְׅ
State of Data Science and
Machine Learning 2021
The 2021 Kaggle DS & ML Survey received 25,973 usable responses from participants in 171 different countries and territories.
Q17. Which of the following ML algorithms do you use on a regular basis? (Select all that apply)

View Slide

הصُ٦ٕٓطحز

View Slide

Object recognition
Game play
ˑ֮׶ָהֲ˒
J’aime la
musique I love music
Speech recognition
Machine translation
Super resolution
3FDBQ
堣唒㷕统כ倜׃ְ ꧟ז
فؚٗٓىؚٝ
فؚٗٓوָع٦س؝٦سׅ׷ךדכזֻծⰅ⳿⸂鋅劤⢽׾׋ֻׁ׿鋅ׇגծ׉ךⰅ⳿⸂׾
ⱄ植ׅ׷ֿהדչفؚٗٓيպ׾欰䧭կ4PGUXBSFծ䗍ⴓ〳腉فؚٗٓىؚٝծFUD

View Slide

p1 p2 p3 p4

ꟼ侧ٌرٕ
Random Forest
Gaussian Process
Logistic Regression
3FDBQ
堣唒㷕统כꟼ侧ٌرٕח״׷ر٦ةⰻ䯏דך✮庠
ꟼ侧ٌرٕفؚٗٓي 㹀纏幥׫ך㛇劤怴皾ךさ䧭ד⡲׸׷Ⰵ⸂̔⳿⸂ךوحؾؚٝ

View Slide

3FDBQ
#SFJNBOךאך侄鎮
Breiman L, Statistical Modeling: The Two Cultures. Statist. Sci. 16(3): 199-231, 2001.
https://doi.org/10.1214/ss/1009213726
Rashomon
Occam
Bellman
葺ְ堣唒㷕统ٌرٕך㢳ꅾ䚍 ꬊ♧䠐䚍

✮庠礵䏝הءٝفׁٕ 鍑ꅸ䚍
ך؝ٝؿؙٔز
넝如⯋䚍כンְַ牜状ַ

View Slide

3FDBQ
如⯋ךンְ
ꟼ侧꬗׾䱿㹀׃׋׶✮庠ָさ׏ג׷ַך嗚鏾׾׃׋׶ׅ׷ךח鋅劤⢽כ⡦挿ֻ׵ְ䗳銲
岣䠐5SBJOJOHח䗳銲זךכ⺡锷ծ7BMJEBUJPOװ5FTUח׮䗳銲礵䏝䱿㹀׮窟鎘涸䱿㹀זךד

View Slide

3FDBQ
如⯋ךンְ
AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTg2JcEd245CGPBAlp64gNpW3aQkTiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVTU21Hca6PmFsfGJyyj8dmJmdmw+GFhbzttGwFJ5TDM2wirJkc03Vec5RHY0XTYtLdVnjBbm2198vNLllq4Z+4LRMXq5LVV09VhXJISp7WhEroQiLMTfCw0D0QARepIzQIw5xBAMKGqiDQ4dDWIMEm74SRDCYxJXRJs4ipLr7HOcIkLZBWZwyJGJr9K/SquSxOq37NW1XrdApGg2LlGFE2Qu7Zz32zB7YK/v8s1bbrdH30qJZHmi5WQleLGc//lXVaXZw8q0a6dnBMbZdryp5N12mfwtloG+edXrZnUy0vcZu2Rv5v2Fd9kQ30Jvvyl2aZ65H+JHJC70YNUj83Y5hkN+IiVuxeDoeSe56rfJjBatYp34kkMQ+UshR/SoucYWO4BdiwqaQGKQKPk+zhB8hJL8AVLKQnA==
x1
AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDQTGuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmM9jzAxOTU94531zc0vLPoDS8t5S2+aMs/JuqqbRUm0uKpoPGcrtsqLhsnFhqTyglTfH+wXWty0FF07tNsGLzfEmqZUFVm0icqeVmKVQIhFmBPBURB1QQhupPTAI45wDB0ymmiAQ4NNWIUIi74SomAwiCujQ5xJSHH2Oc7hI22TsjhliMTW6V+jVcllNVoPalqOWqZTVBomKYMIsxd2z/rsmT2wV/b5Z62OU2PgpU2zNNRyo+K/WM1+/Ktq0Gzj5Fs11rONKnYcrwp5NxxmcAt5qG+ddfvZ3Uy4s8Fu2Rv5v2E99kQ30Frv8l2aZ67H+JHIC70YNSj6ux2jIB+LRLcj8XQ8lNxzW+XFGtaxSf1IIIkDpJCj+jVc4gpdwStEhC0hMUwVPK5mBT9CSH4BVtKQnQ==
x2
x1
x2
x1
x2
x1
x2
x1
x2
x1
x2
N = 32 = 9 N = 52 = 25 N = 102 = 100
x1
x2
AAAChHichVHLTsJAFD3UF+ID1I2JGyLBuDBkUHzEhSG6cclDHgkS0tYBG0rbtIUEiT+gW40LV5q4MH6AH+DGH3DBJxiXmLhx4aU0MUrE20znzJl77pyZKxmqYtmMtT3C0PDI6Jh33DcxOTXtD8zMZi29bso8I+uqbuYl0eKqovGMrdgqzxsmF2uSynNSda+7n2tw01J07cBuGrxYEyuaUlZk0SYq2SwFQizCnAj2g6gLQnAjoQcecYgj6JBRRw0cGmzCKkRY9BUQBYNBXBEt4kxCirPPcQofaeuUxSlDJLZK/wqtCi6r0bpb03LUMp2i0jBJGUSYvbB71mHP7IG9ss8/a7WcGl0vTZqlnpYbJf/ZfPrjX1WNZhvH36qBnm2UseV4Vci74TDdW8g9fePkqpPeToVbS+yWvZH/G9ZmT3QDrfEu3yV56nqAH4m80ItRg6K/29EPsquR6EYkloyF4rtuq7xYwCKWqR+biGMfCWSoPsc5LnApjAorwpqw3ksVPK5mDj9C2PkC2GOP+Q==
y
AAAChHichVHLTsJAFD3UF+ID1I2JGyLBuDBkUHzEhSG6cclDHgkS0tYBG0rbtIUEiT+gW40LV5q4MH6AH+DGH3DBJxiXmLhx4aU0MUrE20znzJl77pyZKxmqYtmMtT3C0PDI6Jh33DcxOTXtD8zMZi29bso8I+uqbuYl0eKqovGMrdgqzxsmF2uSynNSda+7n2tw01J07cBuGrxYEyuaUlZk0SYqWS4FQizCnAj2g6gLQnAjoQcecYgj6JBRRw0cGmzCKkRY9BUQBYNBXBEt4kxCirPPcQofaeuUxSlDJLZK/wqtCi6r0bpb03LUMp2i0jBJGUSYvbB71mHP7IG9ss8/a7WcGl0vTZqlnpYbJf/ZfPrjX1WNZhvH36qBnm2UseV4Vci74TDdW8g9fePkqpPeToVbS+yWvZH/G9ZmT3QDrfEu3yV56nqAH4m80ItRg6K/29EPsquR6EYkloyF4rtuq7xYwCKWqR+biGMfCWSoPsc5LnApjAorwpqw3ksVPK5mDj9C2PkCsAOP5g==
f
AAAChHichVHLTsJAFD3UF+ID1I2JGyLBuDBkUHzEhSG6cclDHgkS0tYBG0rbtIUEiT+gW40LV5q4MH6AH+DGH3DBJxiXmLhx4aU0MUrE20znzJl77pyZKxmqYtmMtT3C0PDI6Jh33DcxOTXtD8zMZi29bso8I+uqbuYl0eKqovGMrdgqzxsmF2uSynNSda+7n2tw01J07cBuGrxYEyuaUlZk0SYqWS4FQizCnAj2g6gLQnAjoQcecYgj6JBRRw0cGmzCKkRY9BUQBYNBXBEt4kxCirPPcQofaeuUxSlDJLZK/wqtCi6r0bpb03LUMp2i0jBJGUSYvbB71mHP7IG9ss8/a7WcGl0vTZqlnpYbJf/ZfPrjX1WNZhvH36qBnm2UseV4Vci74TDdW8g9fePkqpPeToVbS+yWvZH/G9ZmT3QDrfEu3yV56nqAH4m80ItRg6K/29EPsquR6EYkloyF4rtuq7xYwCKWqR+biGMfCWSoPsc5LnApjAorwpqw3ksVPK5mDj9C2PkCsAOP5g==
f
y
x1 AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDQTGuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmM9jzAxOTU94531zc0vLPoDS8t5S2+aMs/JuqqbRUm0uKpoPGcrtsqLhsnFhqTyglTfH+wXWty0FF07tNsGLzfEmqZUFVm0icqeVmKVQIhFmBPBURB1QQhupPTAI45wDB0ymmiAQ4NNWIUIi74SomAwiCujQ5xJSHH2Oc7hI22TsjhliMTW6V+jVcllNVoPalqOWqZTVBomKYMIsxd2z/rsmT2wV/b5Z62OU2PgpU2zNNRyo+K/WM1+/Ktq0Gzj5Fs11rONKnYcrwp5NxxmcAt5qG+ddfvZ3Uy4s8Fu2Rv5v2E99kQ30Frv8l2aZ67H+JHIC70YNSj6ux2jIB+LRLcj8XQ8lNxzW+XFGtaxSf1IIIkDpJCj+jVc4gpdwStEhC0hMUwVPK5mBT9CSH4BVtKQnQ==
x2
y AAAChHichVHLTsJAFD3UF+ID1I2JGyLBuDBkUHzEhSG6cclDHgkS0tYBG0rbtIUEiT+gW40LV5q4MH6AH+DGH3DBJxiXmLhx4aU0MUrE20znzJl77pyZKxmqYtmMtT3C0PDI6Jh33DcxOTXtD8zMZi29bso8I+uqbuYl0eKqovGMrdgqzxsmF2uSynNSda+7n2tw01J07cBuGrxYEyuaUlZk0SYq2SwFQizCnAj2g6gLQnAjoQcecYgj6JBRRw0cGmzCKkRY9BUQBYNBXBEt4kxCirPPcQofaeuUxSlDJLZK/wqtCi6r0bpb03LUMp2i0jBJGUSYvbB71mHP7IG9ss8/a7WcGl0vTZqlnpYbJf/ZfPrjX1WNZhvH36qBnm2UseV4Vci74TDdW8g9fePkqpPeToVbS+yWvZH/G9ZmT3QDrfEu3yV56nqAH4m80ItRg6K/29EPsquR6EYkloyF4rtuq7xYwCKWqR+biGMfCWSoPsc5LnApjAorwpqw3ksVPK5mDj9C2PkC2GOP+Q==
y
y
ֿך⢽דꟼ侧嚊䕎濼׷חכչ挿ֻ׵ְպכ銲׷ַז˘

View Slide

3FDBQ
如⯋ךンְ"QQSPYJNBUJPO5IFPSZך〢Ⱙ涸濼鋅
Bronstein MM, Bruna J, Cohen T, Veličković P.
Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges.
arXiv [cs.LG]. 2021. http://arxiv.org/abs/2104.13478
for all
AAAConichVG7SgNBFD1Z3/EVtRFECEbFKkwkqKQK2ohVHkaFJIbddRKX7IvdSUCDlZ0/YGGlYCF2WvgBNv6ART5BLBVsLLzZLAQN6l1m58yZe+6cmavYuuYKxpoBqae3r39gcCg4PDI6Nh6amNxxrZqj8pxq6Zazp8gu1zWT54QmdL5nO1w2FJ3vKtWN1v5unTuuZpnb4sjmRUOumFpZU2VBVCk0W04UDFkcKkojc7J/EC4IK9whSqEIizIvwt0g5oMI/EhZoQcUcAALKmowwGFCENYhw6UvjxgYbOKKaBDnENK8fY4TBElboyxOGTKxVfpXaJX3WZPWrZqup1bpFJ2GQ8owFtgzu2Fv7Indshf2+Wuthlej5eWIZqWt5XZp/Gw6+/GvyqBZ4LCj+tOzQBlrnleNvNse07qF2tbXj8/fsonMQmORXbFX8n/JmuyRbmDW39XrNM9c/OFHIS/0YtSg2M92dIOd5WhsJRpPxyPJdb9Vg5jBHJaoH6tIYhMp5Kj+KW5wh3tpXtqS0lK2nSoFfM0UvoVU+AKNf5vS
f : Rd ! R
AAACmnichVHLLgRBFD3T3u9BJBIWExOPhUxqRBCrCRtiM8a8EsOkuxUq+pXumgk68wN+wMKKxAIf4ANs/ICFTxBLEhsLd3o6EQS3U12nTt1z61RdzTGEJxl7jChNzS2tbe0dnV3dPb190f6BvGdXXJ3ndNuw3aKmetwQFs9JIQ1edFyumprBC9rBcn2/UOWuJ2wrK48cvmWqe5bYFboqiSpHhw6nDydjJWHFSqYq9zXNz9S2d8rROEuwIGI/QTIEcYSRtqO3KGEHNnRUYILDgiRsQIVH3yaSYHCI24JPnEtIBPscNXSStkJZnDJUYg/ov0erzZC1aF2v6QVqnU4xaLikjGGcPbAr9sLu2Q17Yu+/1vKDGnUvRzRrDS13yn0nwxtv/6pMmiX2P1V/epbYxULgVZB3J2Dqt9Ab+urx6cvGYmbcn2AX7Jn8n7NHdkc3sKqv+uU6z5z94UcjL/Ri1KDk93b8BPmZRHIuMbs+G08tha1qxwjGMEX9mEcKK0gjR/V9nOMaN8qosqSsKmuNVCUSagbxJZTsB6MZmAU=
x, x0 2 Rd
-Lipschitz
AAACqHichVFNLwNRFD0d39/FRmIz0fhM2ryKIFYNGwuL+qgSFZkZr0y8zoyZ16a0fgB/wMKKxEIsLLG28Qcs/ASxJLGxcDudRBDcybx73nn33Dlvru4I05OMPYaUmtq6+obGpuaW1rb2jnBn17Jn512Dpwxb2O6KrnlcmBZPSVMKvuK4XMvpgqf1nZnKebrAXc+0rSW55/D1nLZlmVnT0CRRG+FIOTtUHI7SMjhcVjOC73pCs6Q6p2bKajFaHKRMVSzG/FB/gngAIggiaYevkcEmbBjIIwcOC5KwgAaPnjXEweAQt44ScS4h0z/nOEAzafNUxalCI3aH1i3arQWsRftKT89XG/QVQa9LShX97IFdsBd2zy7ZE3v/tVfJ71HxskdZr2q5s9Fx1LP49q8qR1li+1P1p2eJLCZ9ryZ5d3ymcgujqi/sH78sTi30lwbYGXsm/6fskd3RDazCq3E+zxdO/vCjkxf6YzSg+Pdx/ATLo7H4eGxsfiySmA5G1Yhe9GGI5jGBBGaRRIr6H+IKN7hVRpSkklZWq6VKKNB040so+gci3ZxP
|f(x) f(x0)| 6 Lkx x0k
AAAChHichVG7SgNBFD1ZNcZ31EawCQbFQsJE4wMLCdpYWJjERCGK7K6TOLjZXXYngRj8AW0VCysFC/ED/AAbf8AinyCWEWwsvNksiAb1LrNz5sw9d87M1WxDuJKxekDp6OwKdod6evv6BwaHwsMjOdcqOzrP6pZhOTua6nJDmDwrhTT4ju1wtaQZfFs7Wmvub1e44wrL3JJVm++V1KIpCkJXJVGpjf1wlMWYF5F2EPdBFH5sWuEH7OIAFnSUUQKHCUnYgAqXvjziYLCJ20ONOIeQ8PY5TtBL2jJlccpQiT2if5FWeZ81ad2s6XpqnU4xaDikjGCSPbM71mBP7J69sI9fa9W8Gk0vVZq1lpbb+0OnY5n3f1UlmiUOv1R/epYoYMnzKsi77THNW+gtfeX4spFZTk/WptgNeyX/16zOHukGZuVNv03x9NUffjTyQi9GDYr/bEc7yM3G4guxRCoRTa76rQphHBOYpn4sIol1bCJL9TnOcI4LJajMKHPKfCtVCfiaUXwLZeUTeMOPzA==
L
Donoho DL,
High-dimensional data analysis: The curses and blessings of dimensionality.
Plenary Lecture, AMS National Meeting on Mathematical Challenges of the 21st Century. 2000.
׮׃⡂׋Ⰵ⸂חכ⡂׋⳿⸂׾הְֲ箢ְ鸬竲䚍 -JQTDIJU[鸬竲䚍
׃ַ铬ׁזְ㜥さחכ♧圫ז
ė鵚⡂ח ė
Eؔ٦تך؟ٝفٕ侧ָ䗳銲חז׷կ4PCPMFWDMBTTהַח׃ג׮葺ֻז׵זְկ
⢽Eדչ挿ֻ׵ְպך礵䏝׾実׭׷ז׵Eדכ⭙挿䗳銲דꬊ植㹋涸˘

View Slide

3FDBQ
如⯋ךンְ넝如⯋瑞꟦ךꬊ湫䠬䚍
堣唒㷕统װػة٦ٝ钠陎ך侄猰剅ח״ֻ⳿גֻ׷⢽
猘כ㷕欰儗➿ $#JTIPQ/FVSBM/FUXPSLTGPS1BUUFSO3FDPHOJUJPO
ד剑ⴱח鋅׋

˖ E如⯋瑞꟦חֶֽ׷⽃⡘椔 Sך馄椔
ך⡤琎
AAAC73ichVHNThRBEK4dVFZAWeBi4mXiBrJ72dQSIsTEhOhBj/ztQkLjpqenZ2mZv53p3WSdzAvwAhxMjJB4ID6AD+DFoxcOPILhCMGLidbOTiJKxJ7M9Fdf1VfzdZcVuirWiKcFY+TW7Tujxbtj4xP37k+WpqabcdCNhGyIwA2iLYvH0lW+bGilXbkVRpJ7lis3rb3ng/xmT0axCvwN3Q/ljsfbvnKU4JqoVul1s2JXzacmcyIuEhaqV8kQ2mkyn6bp1YgJO9DsBfc8zlzp6MrVXKTau7qamkwHJpqs0+W2WbGzkCnf0f1qq1TGGmbLvA7qOShDvlaC0idgYEMAArrggQQfNGEXOMT0bEMdEELidiAhLiKksryEFMZI26UqSRWc2D36tinazlmf4kHPOFML+otLb0RKE2bxBI/xHL/gR/yGP/7ZK8l6DLz0abeGWhm2JvcfrH//r8qjXcPub9WNnjU4sJR5VeQ9zJjBKcRQ33tzcL7+ZG02mcMjPCP/h3iKn+kEfu9CfFiVa29v8GORF7oxGlD973FcB835Wv1xbWF1obz8LB9VER7CI6jQPBZhGV7CCjSo/1e4hJ8FMDrGgfHOeD8sNQq5Zgb+WMbxL1/dugg=
V (d) =
⇡ d
2
d
2
· d
2
! 0 (d ! 1)
https://www.math.ucdavis.edu/~strohmer/courses/180BigData/180lecture1.pdf
AAAChHichVHLSsNAFD2Nr/ps1Y3gRiwVF1JutD5wIUU3LltrVdAiSZzW0DQJSVqoxR/QreLClYIL8QP8ADf+gAs/QVxWcOPC2zQgWtQbJnPmzD13zsxVbUN3PaLnkNTR2dXdE+7t6x8YHIpEh0e2XKviaCKnWYbl7KiKKwzdFDlP9wyxYztCKauG2FZLa8397apwXN0yN72aLfJlpWjqBV1TPKYy8n40RgnyY6IdyAGIIYi0Fb3HHg5gQUMFZQiY8BgbUODytwsZBJu5POrMOYx0f1/gGH2srXCW4AyF2RL/i7zaDViT182arq/W+BSDh8PKCcTpiW6pQY90Ry/08Wutul+j6aXGs9rSCns/cjKWff9XVebZw+GX6k/PHgpY8r3q7N32meYttJa+enTRyC5vxOtTdE2v7P+KnumBb2BW37SbjNi4/MOPyl74xbhB8s92tIOt2YS8kEhmkrHUatCqMMYxiWnuxyJSWEcaOa4vcIoznEvd0ow0J823UqVQoBnFt5BWPgE/Y4+x
1
AAACjnichVHNSgJRFD5Of2Y/Wm2CNpIYreQYZhFEUhuX/uQPqMjMeLPBcWaauQomvkD7aBEUBS2iB+gB2vQCLXyEaGnQpkXHcSBKsjPcud/97vnO/e49kqEqFkfsuoSx8YnJKfe0Z2Z2bt7rW1jMWnrDlFlG1lXdzEuixVRFYxmucJXlDZOJdUllOal20N/PNZlpKbp2yFsGK9XFqqYcKbLIiSpgaLNonZi8XemUfQEMoR3+YRB2QACcSOi+RyhCBXSQoQF1YKABJ6yCCBZ9BQgDgkFcCdrEmYQUe59BBzykbVAWowyR2Br9q7QqOKxG635Ny1bLdIpKwySlH4L4gvfYw2d8wFf8/LNW267R99KiWRpomVH2ni2nP/5V1WnmcPytGumZwxFs214V8m7YTP8W8kDfPL3opXdSwfYa3uIb+b/BLj7RDbTmu3yXZKnLEX4k8kIvRg0K/27HMMhuhMLRUCQZCcT2nVa5YQVWYZ36sQUxiEMCMvaLnsMVXAs+ISrsCnuDVMHlaJbgRwjxLwzik/k=
0.5
p
d
AAAChnichVHLSsNAFD3GV62PVt0IbopFcRVupVVxVXTjsg9rC1okidMamiYhSQta/AHBrV24UnAhfoAf4MYfcOEniEsFNy68TQOixXrDZM6cuefOmbmqbeiuR/Q8IA0ODY+MhsbC4xOTU5Ho9MyuazUcTRQ0y7Cckqq4wtBNUfB0zxAl2xFKXTVEUa1tdfaLTeG4umXueMe2KNeVqqlXdE3xmMqTnDqIxkkmP2K9IBGAOILIWNF77OMQFjQ0UIeACY+xAQUuf3tIgGAzV0aLOYeR7u8LnCLM2gZnCc5QmK3xv8qrvYA1ed2p6fpqjU8xeDisjGGRnuiW3uiR7uiFPv+s1fJrdLwc86x2tcI+iJzN5T/+VdV59nD0rerr2UMF675Xnb3bPtO5hdbVN0/ab/mN3GJria7plf1f0TM98A3M5rt2kxW5yz5+VPbCL8YNSvxuRy/YXZETq3Iym4ynN4NWhTCPBSxzP9aQxjYyKHD9Ks5xgbYUkmQpJa11U6WBQDOLHyGlvwBbQpAn
0.5 AAAChnichVHLSsNAFD2Nr1ofrboR3BRLxVWZlmpFEIpuXPZhH1BLSdKxhqZJSNJCLf6A4NYuXCm4ED/AD3DjD7joJ4jLCm5ceJsGRIv1hsmcOXPPnTNzJUNVLJuxnkeYmJyanvHO+ubmFxb9gaXlvKU3TZnnZF3VzaIkWlxVNJ6zFVvlRcPkYkNSeUGqHwz2Cy1uWoquHdltg5cbYk1TThRZtInKVvdilUCIRZgTwVEQdUEIbqT0wCOOUYUOGU00wKHBJqxChEVfCVEwGMSV0SHOJKQ4+xzn8JG2SVmcMkRi6/Sv0arkshqtBzUtRy3TKSoNk5RBhNkLu2d99swe2Cv7/LNWx6kx8NKmWRpquVHxX6xmP/5VNWi2cfqtGuvZxgl2HK8KeTccZnALeahvnXX72d1MuLPBbtkb+b9hPfZEN9Ba7/Jdmmeux/iRyAu9GDUo+rsdoyAfi0S3I/F0PJTcd1vlxRrWsUn9SCCJQ6SQo/o1XOIKXcErRIQtITFMFTyuZgU/Qkh+AeO5kGc=
d = 2 AAAChnichVHLSsNAFD2Nr1ofrboR3BRLxVWZSrUiCEU3LvuwD6ilJOm0hqZJSNJCLf6A4NYuXCm4ED/AD3DjD7joJ4jLCm5ceJsGRIv1hsmcOXPPnTNzJUNVLJuxnkeYmJyanvHO+ubmFxb9gaXlnKU3TZlnZV3VzYIkWlxVNJ61FVvlBcPkYkNSeV6qHw728y1uWoquHdttg5caYk1Tqoos2kRlKvuxciDEIsyJ4CiIuiAEN5J64BEnqECHjCYa4NBgE1YhwqKviCgYDOJK6BBnElKcfY5z+EjbpCxOGSKxdfrXaFV0WY3Wg5qWo5bpFJWGScogwuyF3bM+e2YP7JV9/lmr49QYeGnTLA213Cj7L1YzH/+qGjTbOP1WjfVso4pdx6tC3g2HGdxCHupbZ91+Zi8d7mywW/ZG/m9Yjz3RDbTWu3yX4unrMX4k8kIvRg2K/m7HKMhtRaI7kVgqFkocuK3yYg3r2KR+xJHAEZLIUv0aLnGFruAVIsK2EB+mCh5Xs4IfISS+AOf5kGk=
d = 4
1
AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCKDQTEuDNGNSx7ySJCQtgzYUNqmLSRI/AETt7JwpYkL4wf4AW78ARd8gnGJiRsXXkoTo0S8zXTOnLnnzpm5kqEqls1YzyNMTE5Nz3hnfXPzC4v+wNJyztKbpsyzsq7qZkESLa4qGs/aiq3ygmFysSGpPC/VDwf7+RY3LUXXju22wUsNsaYpVUUWbaIylf1YORBiEeZEcBREXRCCG0k98IgTVKBDRhMNcGiwCasQYdFXRBQMBnEldIgzCSnOPsc5fKRtUhanDJHYOv1rtCq6rEbrQU3LUct0ikrDJGUQYfbC7lmfPbMH9so+/6zVcWoMvLRploZabpT9F6uZj39VDZptnH6rxnq2UcWu41Uh74bDDG4hD/Wts24/s5cOdzbYLXsj/zesx57oBlrrXb5L8fT1GD8SeaEXowZFf7djFOS2ItGdSCwVCyUO3FZ5sYZ1bFI/4kjgCElkqX4Nl7hCV/AKEWFbiA9TBY+rWcGPEBJf6hqQag==
d > 4
AAACjnichVHNSgJRFD5Of2Y/Wm2CNpIYreQoZhFEUhuX/uQPqMjMdLPBcWaauQomvkD7aBEUBS2iB+gB2vQCLXyEaGnQpkXHcSBKsjPcud/97vnO/e49kqEqFkfsuoSx8YnJKfe0Z2Z2bt7rW1jMWXrDlFlW1lXdLEiixVRFY1mucJUVDJOJdUlleam239/PN5lpKbp2wFsGK9fFqqYcKbLIiSpiaKNknZi8HelUfAEMoR3+YRB2QACcSOq+RyjBIeggQwPqwEADTlgFESz6ihAGBIO4MrSJMwkp9j6DDnhI26AsRhkisTX6V2lVdFiN1v2alq2W6RSVhklKPwTxBe+xh8/4gK/4+Wettl2j76VFszTQMqPiPVvOfPyrqtPM4fhbNdIzhyPYsr0q5N2wmf4t5IG+eXrRy2yng+01vMU38n+DXXyiG2jNd/kuxdKXI/xI5IVejBoU/t2OYZCLhMKxUDQVDcT3nFa5YQVWYZ36sQlxSEASsvaLnsMVXAs+ISbsCLuDVMHlaJbgRwiJL6Jhk8c=
0.5
p
2
AAACjnichVHNSgJRFD5Of2Y/Wm2CNpIYreQYZhFEUhuX/uQPqMjMeLPBcWaauQomvkD7aBEUBS2iB+gB2vQCLXyEaGnQpkXHcSBKsjPcud/97vnO/e49kqEqFkfsuoSx8YnJKfe0Z2Z2bt7rW1jMWnrDlFlG1lXdzEuixVRFYxmucJXlDZOJdUllOal20N/PNZlpKbp2yFsGK9XFqqYcKbLIiSpgaLNonZi8XemUfQEMoR3+YRB2QACcSOi+RyhCBXSQoQF1YKABJ6yCCBZ9BQgDgkFcCdrEmYQUe59BBzykbVAWowyR2Br9q7QqOKxG635Ny1bLdIpKwySlH4L4gvfYw2d8wFf8/LNW267R99KiWRpomVH2ni2nP/5V1WnmcPytGumZwxFs214V8m7YTP8W8kDfPL3opXdSwfYa3uIb+b/BLj7RDbTmu3yXZKnLEX4k8kIvRg0K/27HMMhuhMLRUCQZCcT2nVa5YQVWYZ36sQUxiEMCMvaLnsMVXAs+ISrsCnuDVMHlaJbgRwjxLwzik/k=
0.5
p
d
!?
Unit Cube in
Unit Ball
1
AAAChXichVHLSsNAFD2Nr1pfVTeCm2KpuLHcSFFxoUU3LvuwWqhFkjjVYJqEJC1o8QfErbpwpeBC/AA/wI0/4KKfIC4ruHHhbRoQFfWGyZw5c8+dM3NV29Bdj6gZkrq6e3r7wv2RgcGh4ZHo6Nima9UcTRQ0y7Ccoqq4wtBNUfB0zxBF2xFKVTXElnqw1t7fqgvH1S1zwzu0Rbmq7Jl6RdcUj6ncsrwTjVOS/Ij9BHIA4ggiY0XvsY1dWNBQQxUCJjzGBhS4/JUgg2AzV0aDOYeR7u8LHCPC2hpnCc5QmD3g/x6vSgFr8rpd0/XVGp9i8HBYGUOCnuiWWvRId/RM77/Wavg12l4OeVY7WmHvjJxM5N/+VVV59rD/qfrTs4cKFn2vOnu3faZ9C62jrx9dtPJLuURjmq7phf1fUZMe+AZm/VW7yYrc5R9+VPbCL8YNkr+34yfYnEvK88lUNhVPrwatCmMSU5jhfiwgjXVkUOD6FZziDOdSnzQrpaT5TqoUCjTj+BLSygfo4Y/5
> 1
!?

View Slide

3FDBQ
如⯋ךンְ椚锷♳כֲתְֻֻ׻ֽזׁ׉ֲ
΍넝如⯋䚍Ⰵ⸂㢌侧ָ㢳ֺׅ Ύ麓ⶱػًٓة⻉ػًٓة侧ָ㢳ֺׅ
3FT/FU♰ػًٓة
3FT/FU♰ػًٓة
&DJFOU/FU#♰ػًٓة
7((⭙♰ػًٓة
歗⫷׉ךתת׾Ⰵ⸂ׅ׷㜥さ
ؾؙإٕךؕٓ٦歗⫷̔㢌侧
ؾؙإٕךؕٓ٦歗⫷̔♰㢌侧
MBZFS IFBET#&35⭙♰ػًٓة
MBZFS IFBET#&35⭙♰ػًٓة
(159-⭙♰ػًٓة
(15⭙ػًٓة
(PQIFS⭙ػًٓة
Ԩ 堣唒㷕统כⰅ⸂ׁ׸גזְ䞔㜠׾Ⰻֻ
罋䣁׃גֻ׸זְ˘ 亻⡂湱ꟼٔأؙ

Ԩ ה׶ִ֮׆葿ղז㢌侧׾Ⰵ׸ָ׍
歗⫷
鎉铂
植㖈ך堣唒㷕统כչ侧涰♰如⯋ד侧⼪♰ػًٓةךꟼ侧ؿ؍حذ؍ؚٝ׾׃גְ׷պ朐屣ד
➂꟦חה׏גךչؽحؚպر٦ةׅ׵Ⰻֻ駈׶זְכ׆דծ椚锷♳כֲתְֻֻכ׆ָזְ
堣唒㷕统ך剑㣐ךꟼ䗰ז׈ֿך搀椚؜٦鏣㹀ח׮ꟼ׻׵׆ծ׻׶הֲתְֻ׏׍ׯֲך

View Slide

3FDBQ
如⯋ךンְ넝如⯋דⰻ䯏ז׿ג饯ֿ׷׻ֽזְ
Balestriero R, Pesenti J, LeCun Y.
Learning in High Dimension Always Amounts to Extrapolation.
arXiv [cs.LG]. 2021. http://arxiv.org/abs/2110.09485
넝如⯋דכⰻ䯏ז׿ג饯ֿ׷然桦כئٗהְֲطة
넝如⯋ E
דכչⰻ䯏הכ⡦ַպչⰻ䯏ַ㢩䯏ַպ
הְֲ㛇劤涸陽锷ׅ׵㹋כ噰׭גꬊ荈僇
https://youtu.be/86ib0sfdFtw
儗꟦⟃♳ח床׷衼罏㼎锑!.-4USFFU5BML
嗚叨挿
鋅劤挿꧊さ
ך⳻⺪
• "on any high-dimensional (>100) dataset,
interpolation almost surely never happens."
• "Those results challenge the validity of our
current interpolation/extrapolation deﬁnition as
an indicator of generalization performances. "
ⰻ䯏嗚叨挿ָ鎮箺ر٦ة挿ך⳻⺪ח衅׍׋הֹח
ワ׶ך挿ך⦼ַ׵׉ךZ⦼׾寸׭׷ֿה

View Slide

3FDBQ
如⯋ךンְ넝ְ亻⡂湱ꟼٔأؙ
˖ 亻⡂湱ꟼךٔأؙ㣐ֹז㢌侧ف٦ٕ O㢌侧
ַ׵#FTU4VCTFU㔐䌓 N㢌侧
׾䱱ׅה
չ劤䔲כⰋֻ湱ꟼָזְח׮ꟼ׻׵׆պקר䌢ח葺ְ㔐䌓ٌرָٕ鋅אַ׏ג׃תֲ!
˖ ؛ٌ؎ٝؿؓ歲꥔דכꬊ䌢ח〢ַֻ׵濼׵׸גְ׷،٦ثؿ؋ؙز 5PQMJTT

Fan J, Han F, Liu H.
Challenges of Big Data Analysis. Natl Sci Rev. 2014;1: 293–314.
AAACoXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvIaETEG+rEpOSXFOtExKdoxhRn5ir4xadoGOh4asYLKBvoGYCBAibDEMpQZoCCgHyB7QwxDCkM+QzJDKUMuQypDHkMJUB2DkMiQzEQRjMYMhgwFADFYhmqgWJFQFYmWD6VoZaBC6i3FKgqFagiESiaDSTTgbxoqGgekA8ysxisOxloSw4QFwF1KjCoGlw1WGnw2eCEwWqDlwZ/cJpVDTYD5JZKIJ0E0ZtaEM/fJRH8naCuXCBdwpCB0IXXzSUMaQwWYLdmAt1eABYB+SIZor+savrnYKsg1Wo1g0UGr4HuX2hw0+Aw0Ad5ZV+SlwamBs3G454koFuAIQaMIEP06MBkhBnpGZrpmQSaKDs4QaOKg0GaQYlBAxgf5gwODB4MAQyhQPObGFYwbGTYxKTM5MkUwBQEUcrECNUjzIACmKIBAhaZTQ==
(X1, . . . , Xd) ⇠ Nd(0, I)
AAACvnichVFNSxtBGH6y9atp1VgvBS9Lg0VBwqxIbQtCaC89amw0YGSZXSfJ6n51dhK0m/yB/oEeeigVeij+AH+ACJ7aUw/+BPFooZce+u5mQVTUd5mdZ555n3eemdcKXSdSjJ3mtAcDg0PDIw/zjx6Pjo0XJp6sRUFb2qJqB24gaxaPhOv4oqoc5YpaKAX3LFesWztvk/31jpCRE/jv1V4oNj3e9J2GY3NFlFl4VW9xFcuevqTXPb5rxtv1pvgQudxX+nxP7yasakkvtgMpezM105jTa+b2rN41C0VWYmnoN4GRgSKyWA4Kh6hjCwFstOFBwIci7IIjom8DBhhC4jYREycJOem+QA950rYpS1AGJ3aH/k1abWSsT+ukZpSqbTrFpSFJqWOa/WY/2AU7YQfsjP27tVac1ki87NFs9bUiNMc/PV39e6/Ko1mhdam607NCAy9Trw55D1MmuYXd13c+fr5YfV2Zjp+zfXZO/r+xU3ZEN/A7f+zvK6Ly5Q4/FnmhF6MGGdfbcROszZeMF6WFlYVi+U3WqhFM4RlmqB+LKOMdllGl+l9xjJ/4pZW1huZpQT9Vy2WaSVwJbfc/j5mmYg==
ˆ
r = max
j>2
|corr(X1, Xj)|
AAAC8nichVHNSiNBEK6Muv6srlEvgpfBoLggoUeCiiCIXjxqstGAI0PP2Ek6zh89naBO8gK+gKAnlT0su3cfYC97Xzz4CMseXRDBgzU/ICqrNfRU1df1VX/dZfo2DyQhNxmlq7vnQ29f/8DHwaFPw9mR0a3AawqLlS3P9kTFpAGzucvKkkubVXzBqGPabNvcX4v2t1tMBNxzv8hDn+06tObyKreoRMjIOnqdyrDYUZdV3aEHRtgutZcLnTTRTSap0cDUZlXZjlBZF05oeUJ0YmymYmizqh40HSNs6NxVS1icsNSK0dAFr9XlZzXxbSObI3kSm/o60NIgB6lteNkr0GEPPLCgCQ4wcEFibAOFAL8d0ICAj9guhIgJjHi8z6ADA8htYhXDCoroPv5rmO2kqIt51DOI2RaeYuMSyFRhilyTb+SW/CLfyR/y8N9eYdwj0nKI3ky4zDeGj8dLd++yHPQS6k+sNzVLqMJirJWjdj9GoltYCb91dHJbWipOhdPkgvxF/efkhvzEG7itf9bXTVY8e0OPiVrwxXBA2stxvA625vLafL6wWcitrKaj6oMJmIQZnMcCrMA6bEAZ+/+G+4yS6VKkcqqcK5dJqZJJOWPwzJQfj/5suoY=
ˆ
R = max
|S|=4
max
j
corr
0
@X1,
X
j2S
jXj
1
A
0.3 0.4 0.5 0.6 0.5 0.6 0.7 0.8
AAACinichVHLTsJAFL3UFyIK6sbEDZFgXJEpIT43RF245CGPBAhp6wANfaUdSLDhB9y5MpGVJi6MH+AHuPEHXPAJxiUmblx4W5oYJeJtpnPmzD13zswVDUW2GCEDHzc1PTM7558PLAQXl0Lh5ZWCpbdNieYlXdHNkihYVJE1mmcyU2jJMKmgigotiq0jZ7/YoaYl69op6xq0qgoNTa7LksCQKlaaArOzvVo4SuLEjcg44D0QBS/SevgRKnAGOkjQBhUoaMAQKyCAhV8ZeCBgIFcFGzkTkezuU+hBALVtzKKYISDbwn8DV2WP1XDt1LRctYSnKDhMVEYgRl7IPRmSZ/JAXsnnn7Vst4bjpYuzONJSoxa6WMt9/KtScWbQ/FZN9MygDruuVxm9Gy7j3EIa6TvnV8PcfjZmb5Jb8ob+b8iAPOENtM67dJeh2f4EPyJ6wRfDBvG/2zEOCok4vx1PZpLR1KHXKj+swwZsYT92IAUnkIa8W/8SrqHPBbkEt8cdjFI5n6dZhR/BHX8B43OSnw==
ˆ
R
AAACinichVHLTsJAFL3UFyIK6sbEDZFgXJEpIT43RF245CGPBAhp6wANfaUdSLDhB9y5MpGVJi6MH+AHuPEHXPAJxiUmblx4W5oYJeJtpnPmzD13zswVDUW2GCEDHzc1PTM7558PLAQXl0Lh5ZWCpbdNieYlXdHNkihYVJE1mmcyU2jJMKmgigotiq0jZ7/YoaYl69op6xq0qgoNTa7LksCQKlaaArPNXi0cJXHiRmQc8B6IghdpPfwIFTgDHSRogwoUNGCIFRDAwq8MPBAwkKuCjZyJSHb3KfQggNo2ZlHMEJBt4b+Bq7LHarh2alquWsJTFBwmKiMQIy/kngzJM3kgr+Tzz1q2W8Px0sVZHGmpUQtdrOU+/lWpODNofqsmemZQh13Xq4zeDZdxbiGN9J3zq2FuPxuzN8kteUP/N2RAnvAGWuddusvQbH+CHxG94Ithg/jf7RgHhUSc344nM8lo6tBrlR/WYQO2sB87kIITSEPerX8J19DnglyC2+MORqmcz9Oswo/gjr8AJ6KSvw==
ˆ
r
O׾㔐דءىُٖ٦ز

View Slide

3FDBQ
如⯋ךンְ庠䏝ך꧊⚥植韋
˖ 庠䏝ך꧊⚥植韋넝如⯋瑞꟦דכ؟ٝفٕ挿꟦ך騃ꨄָׅץגקה׿וずׄחז׏ג׃תֲ
˖ 騃ꨄ㽯䏝ד䞔㜠ؿ؍ٕةؚٔٝ׾ׅ׷㜥さծ넝如⯋חז׷הקרⰋ嗚稊ח鵚ֻז׷ֿהָ
ر٦ةك٦أװ䞔㜠嗚稊噟歲ד䭷䶯ׁ׸גֹ׋կ
#FZFSך⢽
O⦐ךE如⯋挿
K. Beyer+, When Is “Nearest Neighbor” Meaningful? ICDT’99
V. Pestov, On the geometry of similarity search: dimensionality curse and concentration of measure,
Information Processing Letters, 1999.

View Slide

植㖈ך堣唒㷕统ך⚺ꟼ䗰וֲװ׏ג넝如⯋䚍׾䩛䥿ֽ׷
姻⵱⻉ךرؠ؎ٝ 㽷䨽涸㸜㹀䚍٥أػ٦أ䚍٥鸬竲䚍٥ٗغأز䚍٥FUD

&YQMJDJUז姻⵱⻉׌ֽדכזֻ然桦涸䶏⹛ 4(%瘝
ח״׷*NQMJDJUSFHVMBSJ[BUJPO׮
葺ְ䠬ׄךⴱ劍⦼ךرؠ؎ٝ
㣐鋉垷✲⵸㷕统ך鯄獳זוח״׷葺ְչ8BSN4UBSUպך鏣鎘
ٌرٕ圓鸡װⰅ⸂㢌侧٥Ⰵ⸂邌植װةأؙ圓鸡ךرؠ؎ٝ
帾㾴㷕统ך圓鸡رؠ؎ٝػة٦ٝծ暴䗙ꆀؒٝآص،ؚٔٝծ䎗⡦涸帾㾴㷕统ծ㢳ٌ٦تٕ
ְ׹ְ׹ז灇瑔ָ֮׷˘
ָծ㛇劤涸חכ湡⵸ךةأؙח״ֻوحثׅ׷չ葺ְ䌓秛غ؎،أպךرؠ؎ٝך㉏겗
넝ְ荈歋䏝׾䭯א堣唒㷕统ٌرָٕ♶鿪さ٥♶黝䔲זꟼ侧׾䠐㔳ׇ׆邌植׃ג׃ת׻זְ
״ֲٌرٕ瑞꟦װ㷕统倯䒭װٌرٕ圓鸡׾ⵖꣲ٥ⵖ秈٥ⵖ䖴ׅ׷ 䠐㔳涸غ؎،أ׾铬ׅ

Inductive Bias

View Slide

ر٦ة Decision
Tree
Random
Forest GBDT
Nearest
Neighbor
Logistic
Regression
SVM
Gaussian
Process
Neural
Network
1JFDFXJTFDPOTUBOU
3FDBQ寸㹀啾ה剑鵚ꦄ岀1JFDFXJTFDPOTUBOUQSFEJDUPST
1JFDFXJTFMJOFBS

View Slide

Journal of the American Statistical Association , Jun., 2006, Vol. 101, No. 474 (Jun., 2006), pp. 578-590
https://www.jstor.org/stable/27590719
"We introduce a concept of potential nearest neighbors (k-PNNs) and show that random
forests can be viewed as adaptively weighted k-PNN methods. "

View Slide

˖ ٓٝتيطأ
˖ ،ٝ؟ٝـٕ
ָ꒲חז׏גְ׷
醱꧟זꟼ侧ח厫鮾ח
ؿ؍حزׅ׷׋׭חכ
⸇ִג

View Slide

3FDBQ寸㹀啾ה剑鵚ꦄ岀
剑鵚ꦄ岀ז׿ג˘ה䙼ֲה䙼ְתָׅծ،مד׮ⴓַ׷知稆ׁה
㹋欽䚍ꬊ䌢ח⮚׸׋椚锷涸䚍颵׾Ⱟי⪒ִגְ׷˘
The Bible or "The Yellow Terror"
by Luc, Laci, Gábor
4UPOFךِصغ٦؟ٕ♧荜䚍㹀椚
ך䕦갟ד剑鵚ꦄ岀כ䎃➿ך
ظٝػًٓزٔحؙ窟鎘ך椚锷؝ىُصذ؍ך♧㣐ꟼ䗰✲

View Slide

剑鵚ꦄ岀ז׿ג˘ה䙼ֲה䙼ְתָׅծ،مד׮ⴓַ׷知稆ׁה
㹋欽䚍ꬊ䌢ח⮚׸׋椚锷涸䚍颵׾Ⱟי⪒ִגְ׷˘
4UPOFךِصغ٦؟ٕ♧荜䚍㹀椚
ך䕦갟ד剑鵚ꦄ岀כ䎃➿ך
ظٝػًٓزٔحؙ窟鎘ך椚锷؝ىُصذ؍ך♧㣐ꟼ䗰✲
〢Ⱙ涸ז䱿庠窟鎘㷕 ػًٓزٔحؙ窟鎘

؜٦ي鏣㹀錁庠⢽ך⦼׌ַֽ׵欰䧭㐻ךػًٓة⦼ ĞהĤ
׾䔲ג׵׸׷
AAACnHichVHLSsNAFL2Nr1ofrbpQEKRYlAoaboovBEHUhSCKr7aCLSWJ0zqYF0laqMUf8AdcuFJxIeraD3DjD7jwE8SlghsX3qYB0WK9YTJnztxz58xcxdK44yI+B4Sm5pbWtmB7qKOzqzsc6elNOWbRVllSNTXT3lVkh2ncYEmXuxrbtWwm64rG0srhUnU/XWK2w01jxy1bLKvLBYPnuSq7ROUi/evxjF6cn5DExHg04/CCLs8nxKmxXCSGInoRrQeSD2Lgx4YZuYcM7IMJKhRBBwYGuIQ1kMGhbw8kQLCIy0KFOJsQ9/YZHEOItEXKYpQhE3tI/wKt9nzWoHW1puOpVTpFo2GTMgoj+ITX+IaPeIMv+PlnrYpXo+qlTLNS0zIrFz4Z2P74V6XT7MLBt6qhZxfyMOt55eTd8pjqLdSavnR0+rY9tzVSGcULfCX/5/iMD3QDo/SuXm2yrbMGfhTyQi9GDZJ+t6MepBKiNC1Obk7GFhb9VgVhEIYhTv2YgQVYgQ1Ier4u4RbuhCFhWVgV1mqpQsDX9MGPEFJfeQKW8A==
N(µ = 1.2, = 2.5)
✉侧欰䧭

欰䧭㐻ךչ㘗պ 姻鋉ⴓ䋒הַ
׾⟎㹀׃זְ

View Slide

Given , a real i.i.d. sequence, estimate
跐駈( CPS-VHPTJ
NeurIPS 2021 Invited Talk (Breiman Lecture)
Do we know how to estimate the mean?
׋׌JJE然桦㢌侧ך䎂㖱⦼׾䱿㹀ׅ׷鑧

AAACnHichVG7SgNREJ2s7/hI1EJBkGBQLCTMRo3GStRCECE+EgNJCLubm7i4L3ZvAhr8AX/AwkrFQtTaD7DxByzyCWKpYGPh7GZFLBJnuXtnzsyZe+4d2dJUhyM2AkJHZ1d3T29fsH9gcCgUHh7JOGbVVlhaMTXTzsqSwzTVYGmuco1lLZtJuqyxA/lo3c0f1JjtqKaxz48tVtCliqGWVUXiBBXDY3lZr2dPi+JcvmRyZ84PKRPFGCYWk/MYwdgiikvJJDmIieX5eEQkx7Uo+JYyw4+QhxKYoEAVdGBgACdfAwkc+nIgAoJFWAHqhNnkqV6ewSkEiVulKkYVEqFH9K9QlPNRg2K3p+OxFTpFo2UTMwLT+IK3+I7PeIev+NWyV93r4Wo5pl1ucplVDJ2N733+y9Jp53D4y2qrmUMZlj2tKmm3PMS9hdLk107O3/dWdqfrM3iFb6T/Ehv4RDcwah/KzQ7bvWijRyYt9GI0oJ8pRFo7mXhMTMQWdhaiq2v+qHphAqZgluaxBKuwCSlIe7qu4R4ehElhQ9gStpulQsDnjMIfEzLfLdyZlw==
X1, . . . , Xn
AAACo3ichVFNSxtRFD1O/aq2JtqNUJDBkOIq3PEjmkJBLAXBjUajARPCvPGpg/PFzEvADtl15R/owlULXbQuW/AHuPEPuPAniEsFNy56ZxIpLrR3ePPOO++e+857VwSOHSmiyx7tRW9f/8Dgy6HhV69HMtnRsc3Ib4aWrFi+44dVYUbSsT1ZUbZyZDUIpekKR26Jg4/J/lZLhpHtexvqMJB119zz7F3bMhVTjexETbhxzW229Q96zTXVvhDxp7aesNV2w2hkc1Sg4lxphnQqzJExXyoxICouzEzrBoMkcujGqp89RQ078GGhCRcSHhRjByYi/rZhgBAwV0fMXMjITvcl2hhibZOzJGeYzB7wf49X213W43VSM0rVFp/i8AhZqSNPF/STbuicTuiK7p+sFac1Ei+HPIuOVgaNzNH4+t1/VS7PCvv/VM96VtjFQurVZu9ByiS3sDr61uevN+vvy/n4HX2na/b/jS7pjG/gtW6tH2uyfPyMH8Fe+MW4QQ9d0J8Gm9MFo1iYXZvNLS51WzWIt5jEFPdjHotYxioqXP8LfuE3/mh5bUUraxudVK2nq3mDR6HV/wJh+5ws
µ = EX1
״׶葺ְך֮׷ך
AAACqHichVG7ThtBFD1sSHjkgZM0SDQrLBBJ4dzlYTASEgpNSvMwdoSJtbsZYMS+tDu2MCt/APwARapEokApKIGaJj9AwScgSpBoKHJ37ShKAbmr2Tlz5p47Z+ZagSMjRXTZpT3pfvqsp7ev//mLl68GMq/frEZ+PbRFyfYdP6xYZiQc6YmSksoRlSAUpms5omxtLyT75YYII+l7K6oZiHXX3PTkhrRNxVQtk60qsaMi1XTEWDWqu7VYzhmtL55etdy40qrJdx+SLMpRfqowQTrlpsiYLhQYEOVnJsZ1g0ESWXSi6GdOUMVX+LBRhwsBD4qxAxMRf2swQAiYW0fMXMhIpvsCLfSzts5ZgjNMZrf5v8mrtQ7r8TqpGaVqm09xeISs1DFCF3REN/SLftIV3T9YK05rJF6aPFttrQhqA/uDy3f/Vbk8K2z9VT3qWWEDM6lXyd6DlEluYbf1jd2Dm+XZpZF4lH7QNfv/Tpd0zjfwGrf24aJY+vaIH4u98Itxg/50QX8YrI7njHxucnEyO/+x06peDGEYY9yPaczjE4oocf09HOMUZ9p7raiVtc/tVK2ro3mLf0KzfgMt4J6c
(
Pn
i=1
Xi)/n
،ٝ؟ٝـٕה
㺘זꟼ⤘

View Slide

Journal of Machine Learning Research, 9, 2015-2033, 2008 The Annals of Statistics, 43(4), 1716-1741, 2015.

View Slide

הصُ٦ٕٓطحز

View Slide

3FDBQ
#SFJNBOךאך侄鎮
https://doi.org/10.1214/ss/1009213726
Rashomon
Occam
Bellman
葺ְ堣唒㷕统ٌرٕך㢳ꅾ䚍 ꬊ♧䠐䚍

✮庠礵䏝הءٝفׁٕ 鍑ꅸ䚍
ך؝ٝؿؙٔز
넝如⯋䚍כンְַ牜状ַ

View Slide

3BTIPNPO&GGFDU
https://doi.org/10.1214/ss/1009213726
• "What I call the Rashomon Effect is that there is often a multitude of different descriptions
[equations f︎(x)] in a class of functions giving about the same minimum error rate. The most
easily understood example is subset selection in linear regression."
• "The Rashomon Effect also occurs with decision trees and neural nets."
• "This effect is closely connected to what I call instability (Breiman,1996a) that occurs when
there are many different models crowded together that have about the same training or test
set error."
Rashomon Effect: 葺ְ堣唒㷕统ٌرٕך㢳ꅾ䚍 ꬊ♧䠐䚍

堣唒㷕统؝ٝلדכ♧אךر٦ةإحزח㼎׃ג㢳圫זٌرָٕ䲿⳿ׁ׸׷կ
✮庠礵䏝כ♳⡘כ׌׿׀朐䡾חז׶㹋欽♳כקרず瘝ח葺ְٌرٕה׫זׇ׷կ
⢽

View Slide

3BTIPNPO&GGFDUה6OEFSTQFDJGJDBUJPO
穠㽷ծ鎮箺ر٦ة׮ 嗚鏾ر٦ة׮
ذأزر٦ة׮剣ꣲ׌ֽוչ넝如⯋瑞꟦պדכ
溪ךꟼ侧ָ֮׷ה׃ג׮鋅劤侧♶駈ד׉׸׾暴㹀 TQFDJGZ
׃ֹ׸זְַ׵
׮׃؟ٝفٕ侧ָ⼧ⴓז׵植➿涸זꬊ简䕎䩛岀ז׵׌ְ׋ְו׸׾鼅׿ד׮0,זכ׆
Kernel Ridge (RBF)
Neural Network (MLP)
Gradient Boosted Trees
SVR (RBF) Gaussian Process (RBF)
Random Forest
Nearest Neighbors Decision Tree

View Slide

3BTIPNPO&GGFDUה6OEFSTQFDJGJDBUJPO
穠㽷ծ鎮箺ر٦ة׮ 嗚鏾ر٦ة׮
ذأزر٦ة׮剣ꣲ׌ֽוչ넝如⯋瑞꟦պדכ
溪ךꟼ侧ָ֮׷ה׃ג׮鋅劤侧♶駈ד׉׸׾暴㹀 TQFDJGZ
׃ֹ׸זְַ׵
6OEFSTQFDJFEז朐屣ך㜥さծ欽ְ׷䩛岀װع؎ػ٦ػًٓةח״׏ג׌ְע殯ז׷穠卓ח
Kernel Ridge (RBF)
Random Forest

View Slide

https://arxiv.org/abs/2011.03395
https://ai.googleblog.com/2021/10/
how-underspecification-presents.html
"While ML models are validated on held-out data, this validation is often insufficient to
guarantee that the models will have well-defined behavior when they are used in a new setting. "
6OEFSTQFDJGJDBUJPOכؽحؚر٦ة✲⢽ד׮饯ֿ׏גְ׷
չ葺ְ侄䌌ٓكָٕאֽ׵׸׋㣐鋉垷ر٦ةך㹋⢽ד׮պ饯ֿ׏גְ׷ךַ׮

View Slide

罋ִ׋ְ挿΍
Kernel Ridge (RBF)
Random Forest
չ寸㹀加،ٝ؟ٝـٕדכ㔐䌓刼简ת׻׶ד㔐䌓⦼ָע׸װְׅպכوؤְ暴䚍ַ
̔׮ה׮הך؟ٝفٕ⦼ָ溪ך⫘ぢַ׵ٓٝتي㢌⹛׃גְ׷ך׌ַ׵ֿך玎䏝✮庠⦼ָ䮶׸׷
խךכ鷞ח⨳Ⰻדכչ✮庠ⴓ侔պ׾鎘皾׃ג➰♷ׅ׸ל׬׃׹㹋欽欽鷿ד׮(PPEזכ׆

View Slide

罋ִ׋ְ挿Ύ
չ寸㹀加כ0WFSUUJOH׃װְׅպכوؤְ暴䚍ַ
̔،ٝ؟ٝـٕ㷕统פך⚺⹛堣חז׏גְ׷կ׋׌׃ծֿךչ0WFSUUJOHպכ䗳׆׃׮剣㹱הכ
鎉ִזְ #FOJHOPWFSUUJOH
կظ؎ؤ֮׶✲⢽ד鎮箺铎䊴ד׮׉׸ז׶ח䕵ח甧א
Kernel Ridge (RBF)
Random Forest
Benignʁ
Benignʁ Benignʁ
Malignant Malignant

View Slide

罋ִ׋ְ挿΍寸㹀啾㔐䌓ך✮庠ⴓ侔 ⥋걾⼒꟦
ך䱿㹀
㔐䌓׾㹋ꥷח䠐䙼寸㹀ח崞欽ׅ׷ꥷחכծ✮庠⦼׌ֽדזֻ׉ך⥋걾䏝 ♶然㹋䚍

ך䞔㜠ָהג׮ꅾ銲
"ֿך㉀ㅷך㡰׶♳־✮庠⦼כ p
דׅ
#ֿך㉀ㅷך㡰׶♳־✮庠⦼כ p
דׅ
Ԩ ✮庠⦼ךⴓ侔ִׁ皾⳿דֹ׸ל 姻鋉鵚⡂ד
⥋걾⼒꟦װ鷵如㹋꿀鎘歗ח崞欽ׅ
׷劍䖉⦼何㊣ꆀ &YQFDUFE*NQSPWFNFOU
זוךꆀ׮鎘皾דֹ׷
⥋걾⼒꟦

View Slide

罋ִ׋ְ挿΍寸㹀啾㔐䌓ך✮庠ⴓ侔 3BOEPN'PSFTU㘗

Ԩ 㔐䌓加⽃⡤ד荈搫ז✮庠ⴓ侔׾䭯א
Ը✮庠挿ָ衅׍׋걄㚖ח֮׷鎮箺؟ٝفٕךZךչⴓ侔պ
Ԩ 3BOEPN'PSFTU㘗ך㜥さכ寸㹀加ָ㛇劤涸ח杝甧זךדծぐղך㔐䌓加ך✮庠
ⴓ侔׾窟さ׃ג،ٝ؟ٝـٕך✮庠ⴓ侔׾皾⳿
Ⰻⴓ侔ך岀⵱ -BXPGUPUBMWBSJBODF

Ⰻⴓ侔ؚٕ٦فⰻⴓ侔ؚٕ٦ف㢩ⴓ侔
Hutter et al, Algorithm Runtime Prediction: Methods & Evaluation (2014) https://arxiv.org/abs/1211.0906

View Slide

罋ִ׋ְ挿΍寸㹀啾㔐䌓ך✮庠ⴓ侔 3BOEPN'PSFTU㘗

https://scikit-optimize.github.io/stable/_modules/skopt/learning/forest.html
def _return_std(X, trees, predictions, min_variance):
# This derives std(y | x) as described in 4.3.2 of arXiv:1211.0906
std = np.zeros(len(X))
for tree in trees:
var_tree = tree.tree_.impurity[tree.apply(X)]
# This rounding off is done in accordance with the
# adjustment done in section 4.3.3
# of http://arxiv.org/pdf/1211.0906v2.pdf to account
# for cases such as leaves with 1 sample in which there
# is zero variance.
var_tree[var_tree < min_variance] = min_variance
mean_tree = tree.predict(X)
std += var_tree + mean_tree ** 2
std /= len(trees)
std -= predictions ** 2.0
std[std < 0.0] = 0.0
std = std ** 0.5
return std
Hutter et al, Algorithm Runtime Prediction: Methods & Evaluation (2014) https://arxiv.org/abs/1211.0906

View Slide

罋ִ׋ְ挿΍寸㹀啾㔐䌓ך✮庠ⴓ侔 ⥋걾⼒꟦
ך䱿㹀
⥋걾⼒꟦
RandomForestRegressor
(n_estimators=10, max_leaf_nodes=12)
ExtraTreesRegressor
(n_estimators=10, max_leaf_nodes=32)
ֿ׸הずׄ倯岀׾׉ךתת(SBEJFOU#PPTUJOHך寸㹀加꧊さח黝欽ׅ׷ה˘
GradientBoostingRegressor
(n_estimators=30, learning_rate=0.1)
ז׿ַقٝ
ת׋(SBEJFOU#PPTUJOHכ㷕统桦ח״׷ꅾ׫➰ֹ
䎂㖱זךדꅾ׫ך罋䣁׮䗳銲

View Slide

罋ִ׋ְ挿΍寸㹀啾㔐䌓ך✮庠ⴓ侔 (SBEJFOU#PPTUJOH㘗

Ԩ (SBEJFOU#PPTUJOHדכ䴦㣟ꟼ侧׾荈歋ח㢌ִ׵׸׷挿ח滠湡׃ג鸐䌢כⴓ⡘挿
㔐䌓 2VBOUJMF3FHSFTTJPO
ד✮庠ⴓ侔湱䔲ꆀ׾皾⳿ׅ׷կ
Ԩ 姻鋉ⴓ䋒ׅ׷ر٦ةך㜥さծ垥彊⨉䊴ח湱䔲ׅ׷ⴓ⡘挿כ ♴⩎
ה ♳
⩎
חז׷ךדծֿךⴓ⡘挿ח㼎׃ג㔐䌓ׅ׸ל垥彊⨉䊴ך♳⩎ה♴⩎ך⦼ָ䖤
׵׸׷
Ԩ ⴓ⡘挿㔐䌓 2VBOUJMF3FHSFTTJPO

GradientBoostingRegressor(loss='quantile', alpha=α)
ر٦ةך溪׿⚥ 䎂㖱⦼
ד
כזֻ暴㹀ך醣 Rⴓ⡘挿

׾湫䱸杆׏׋㔐䌓

View Slide

罋ִ׋ְ挿΍寸㹀啾㔐䌓ך✮庠ⴓ侔 (SBEJFOU#PPTUJOH㘗

Ԩ 䴦㣟ꟼ侧׾ⴓ⡘挿ٗأ 2VBOUJMFMPTT
ח㢌刿׃ĥ ךⴓ⡘挿R׾
✮庠ׅ׷㔐䌓׾׉׸׊׸遤ֲկ
GradientBoostingRegressor(
(n_estimators=30, learning_rate=0.1)
LGBMRegressor (n_estimators=30, learning_rate=0.1,
max_leaf_nodes=8, min_child_samples=5)
Ԩ ⴓ⡘挿ָ䎂㖱⦼ծ։ך꟦ָ垥彊⨉䊴ה鵚⡂דֹ׷ךדծ⥋걾⼒꟦װ
劍䖉⦼何㊣ꆀזו׮皾⳿דֹ׷
aka "Pinball loss"

View Slide

剑ⴱך⢽
Rando Forest ExtraTrees (w/o bootstrap)
ExtraTrees (w/ bootstrap)
Rando Forest ExtraTrees (w/o bootstrap)
ExtraTrees (w/ bootstrap)

View Slide

׍ז׫ח˘
׉׮׉׮ֿ׸כ⳿勻ג葺ְךַ
ֿ׸ז׵ה׮ַֻ˘
ֿ׸כ➂䊨ر٦ةד姻鍑ָ֮׷ر٦ة
זךד
׉ך䠐㄂דכֿ׸ךקֲָ葺ְהכ鎉ִ
ֿ׸ד׮荈搫ז孡׮ׅ׷˘
崞䚍⻉ꟼ侧׌ֽ
3F-6̔5BOIח㢌刿

View Slide

׍ז׫ח̕ך״ֲז׉׮׉׮锷׮㣐ְח֮׷˘
ֿ׸ז׵ה׮ַֻ˘ ׉׮׉׮锷ה׃גծ窟鎘㷕涸חכ؟ٝفٕ♶駈ד
֮׷6OEFSTQFDJFEז朐屣ד⡦ַ䒉鏣涸ז陽锷
כ〳腉זךַ
堣唒㷕统ٌرٕךEFQMPZ䖓ח⳿⠓ֲر٦ة ذأ
زر٦ة
כ搀ꣲծ鎮箺ر٦ة٥嗚鏾ر٦ةכ剣
ꣲծהְֲ搀椚鏣㹀דכ穠㽷չٌرٕך䌓秛غ
؎،أ JOEVDUJWFCJBT
պָ湡⵸ך㉏겗חوحث
ׅ׷ַ׌ָֽꅾ銲
ֿך䠐㄂ד׮ٌرٕך䮙⹛ך帾ְ椚鍑ה
׉׸׾㹋欽خ٦ٕה㹋㉏겗ח鼧⯋ׅ׷
㹋騧ךJUFSBUJPOכ㣐✲

View Slide

ك؎ؤדִִװ׿˘הְֲ䠐鋅כ֮׷ה䙼ֲֽו˘
Gaussian Process Gaussian Process
؟ٝفָٕ׋ֻׁ׿֮׷ה
✮庠ⴓ侔קרחז׶ָ׍
⽃ז׷3#'װ.BUFSOדכ
搀椚
ְ׆׸חׇ״⡦׵ַךꬊ简䕎ٌرؚٔٝכ䗳銲կ⢽ִלծؕ٦طٕ岀ד׉׸׾遤ֲؖؐأ麓玎
㔐䌓ד葺ְ䠬ׄחׅ׷ךכַז׶耵➂涸זؕ٦طٕإָٔؗ銲׷✮䠬˘
猘䠬׮׃ؕ٦طٕ岀ד׮ֲתֻ⳿勻׷ז׵ֿ׸׌ֽ植㜥ד寸㹀加،ٝ؟ٝـָٕꅾ㹇ׁ׸׷
ֿה׮זְךדכהְֲ孡׮ׅ׷կ植➿涸ر٦ة׾ؕ٦طٕ岀דֲתֻ䪔ֲחכ耵➂䪮ָ䗳銲

View Slide

/FVSBM/FUXPSLT .-1
⥋걾⼒꟦䱿㹀
MLP
(Quantile Regression)
MLP
(Quantile Regression)
MLP
(MC Dropout)
MLP
(MC Dropout)
arch:
(Linear(1, 100), ReLU,
Dropout(p),
Linear(100, 100), ReLU,
Dropout(p),
Linear(100, 1))
p=0.25
#sampling=30
p=0.25
#sampling=30
p=0.0
p=0.0
׍׳׏ה㢌ז䠬ׄ
%SPQPVUךְׇד
畭׏ֿך⦼כוֲ
׃ג׮ⴓ侔㣐ֹ׭ח

View Slide

MLP
(30 Ensemble)
MLP
(30 Ensemble)
p=0.0
p=0.0
MLP
(30 Ensemble)
MLP
(30 Ensemble)
p=0.25
p=0.25
%SPQPVUךְׇד
畭׏ֿך⦼כוֲ
׃ג׮ⴓ侔㣐ֹ׭ח
/FVSBM/FUXPSLT .-1
⥋걾⼒꟦䱿㹀
arch:
(Linear(1, 100), ReLU,
Dropout(p),
Linear(100, 100), ReLU,
Dropout(p),
Linear(100, 1))

View Slide

.-1כ׍ׯ׿ה⢪ִל殯圫חػٙؿٕ㤴ָ帾ְ˘
• Tolstikhin IO, Houlsby N, Kolesnikov A, Beyer L, Zhai X, Unterthiner T, et al.
MLP-Mixer: An all-MLP Architecture for Vision. NeurIPS 2021.
• Kadra A, Lindauer M, Hutter F, Grabocka J.
Well-Tuned Simple Nets Excel on Tabular Datasets. NeurIPS 2021.
• Liu H, Dai Z, So D, Le Q.
Pay Attention to MLPs. NeurIPS 2021.
• Melas-Kyriazi L.
Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly
Well on ImageNet. 2021. http://arxiv.org/abs/2105.02723
׍ׯ׿ה׃׋ر٦ةד㷕统׮黝ⴖח؝ٝزٗ٦ׁٕ׸גְ׸ל$//׮5SBOTGPSNFS
׮"UUFOUJPOׅ׵׮銲׵זְ׿ַ׮˘ .-1׾ֲתֻ⢪ִל״ְ׌ֽזךַ׮

View Slide

罋ִ׋ְ挿Ύ
չ寸㹀加כ0WFSUUJOH׃װְׅպכوؤְ暴䚍ַ
̔،ٝ؟ٝـٕ׾《׷⚺⹛堣חז׏גְ׷կ׋׌׃ծֿךչ0WFSUUJOHպכ䗳׆׃׮剣㹱הכ
鎉ִזְ #FOJHOPWFSUUJOH
կظ؎ؤ֮׶✲⢽ד鎮箺铎䊴ד׮׉׸ז׶ח䕵ח甧א
Kernel Ridge (RBF)
Random Forest
Benignʁ
Benignʁ Benignʁ
Malignant Malignant

View Slide

PolyReg(1)
RMSE 0.299
PolyReg(3)
RMSE 0.28
PolyReg(5)
RMSE 0.225
PolyReg(7)
RMSE 0.113
PolyReg(10)
RMSE 0.0189
PolyReg(15)
RMSE 0.00737
PolyReg(20)
RMSE 0.000
PolyReg(30)
RMSE 0.000
ExtraTrees (no bootstrap)
RMSE 0.000
ExtraTrees (bootstrap)
RMSE 0.0121
Random Forest
RMSE 0.012
LGBM
RMSE 0.0508
95%-CI 95%-CI 95%-CI 95%-CI
Problematic overﬁtting by polynomial regression of order k
clearly overﬁtted but harmless (still informative)
also we can assess
the uncertainty
#FOJHO0WFSGJUUJOHظ؎ؤ֮׶ر٦ةד鎮箺铎䊴ד׮搀㹱

View Slide

&YUSB5SFFTװ(SBEJFOU#PPTUFE5SFFTכ鎮箺铎䊴כ⡭酔
Ⰻֻٓٝتيזٓكٕד׮鎮箺铎䊴׾麦䧭דֹ׷
ExtraTrees (no bootstrap) ExtraTrees (no bootstrap) ExtraTrees (no bootstrap) ExtraTrees (no bootstrap)
Gradient Boosted Trees Gradient Boosted Trees Gradient Boosted Trees Gradient Boosted Trees

View Slide

׍ז׫ח剑鵚ꦄ岀װ寸㹀加׮䭯א䚍颵ד׻׶ה䔲׋׶⵸
Nearest Neighbor (k=1) Nearest Neighbor (k=1) Nearest Neighbor (k=1) Nearest Neighbor (k=1)
Decision Tree Decision Tree Decision Tree Decision Tree
Ⰻֻٓٝتيזٓكٕד׮鎮箺铎䊴׾麦䧭דֹ׷ ֿך鏣㹀דכ(#%5 // %5כקרずׄ

View Slide

3BOEPN'PSFTUװ剑鵚ꦄ岀כֿך䚍颵׾䭯׋זְ
Random Forest Random Forest Random Forest Random Forest
Nearest Neighbor (k=3) Nearest Neighbor (k=3) Nearest Neighbor (k=3) Nearest Neighbor (k=3)
#PPUTUSBQ׾⠵ֲ3BOEPN'PSFTUװ&YUSB5SFFTծL// L
זוכPWFSUדֹזְ

View Slide

帾㾴㷕统ٌرָֿٕך䚍颵׾䭯אֿהכ岣湡ׁ׸גְ׷
"our experiments establish that state-of-the-art convolutional networks for image classiﬁcation trained with
stochastic gradient methods easily ﬁt a random labeling of the training data. This phenomenon is qualitatively
unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured
random noise. "
Berner J, Grohs P, Kutyniok G, Petersen P.
The Modern Mathematics of Deep Learning. arXiv [cs.LG]. 2021. http://arxiv.org/abs/2105.04026
Zhang C, Bengio S, Hardt M, Recht B, Vinyals O.
Understanding Deep Learning (Still) Requires Rethinking Generalization. Commun ACM. 2021;64: 107–115.

View Slide

#FOJHO0WFSGJUUJOH
Berner J, Grohs P, Kutyniok G, Petersen P.
The Modern Mathematics of Deep Learning. arXiv [cs.LG]. 2021. http://arxiv.org/abs/2105.04026
穗꿀涸✲㹋ה׃גծ㹋ꥷך帾㾴㷕统ך植㜥דכ侄䌌ٓكٕחظ؎ؤָ֮׹ֲָזַ׹ֲָծ
ذأز铎䊴ָ㼭ְׁٌرٕכչ鎮箺铎䊴׮㼭ְׁ קרئٗ铎䊴
պז㜥さָהג׮㢳ְկ

View Slide

PS%PVCMF%FTDFOU

View Slide

*OUFSQPMBUJPO 1FSGFDU'JUUJOH BOE%PVCMF%FTDFOU
PNAS (2020) Ann. Statist. (2020)
NeurIPS (2018) arXiv (2019)

View Slide

#FOJHO0WFSGJUUJOH 猘䠬

⿫罋
➙屭⯇耆帾㾴㷕统ך⾱椚鍑匿害⻉铎䊴ך⩎꬗ַ׵傈劤窟鎘㷕⠓钞

View Slide


˖ صُ٦ٕٓطحزװ寸㹀加דכչ⡚如⯋ד׮պظ؎ؤ֮׶ر٦ةד鎮箺铎䊴קרך㜥さָ
剑׮ذأز铎䊴׮㼰זְ㹋⢽ח穠圓⳿⠓ֲךדծず颵ד䪔׏ג葺ְךַכ㣐ְח毟㉏կ
⿫罋

View Slide


˖ הְֲַչ#FOJHO0WFSUUJOHպչ%PVCMFEFTDFOUպזךַכַז׶毟㉏կ׉׮׉׮
%PVCMFEFTDFOUדכ♧㔐ذأز铎䊴ָ䝤ֻז׷غٝفָ֮׷ֿהחז׷ָծ"EBCPPTUךⴱ劍
陽锷׾瘗걧ח鎮箺铎䊴ך䖓׮ 䝤ֻז׷䴎׶䨱׃כ暴ח搀׃ח
ذأز铎䊴ָ幾׶אבֽ׷✲
⢽כ葿ղ㜠デ׮֮׷կ
⿫罋

View Slide


˖ הְֲַչ#FOJHO0WFSUUJOHպչ%PVCMFEFTDFOUպזךַכַז׶毟㉏կ׉׮׉׮
%PVCMFEFTDFOUדכ♧㔐ذأز铎䊴ָ䝤ֻז׷غٝفָ֮׷ֿהחז׷ָծ"EBCPPTUךⴱ劍
陽锷׾瘗걧ח鎮箺铎䊴ך䖓׮ 䝤ֻז׷䴎׶䨱׃כ暴ח搀׃ח
ذأز铎䊴ָ幾׶אבֽ׷✲
⢽כ葿ղ㜠デ׮֮׷կ
˖ -JOFBS3FHSFTTJPO 3JEHFMFTT-FBTUTRVBSFT
ד׮饯ֿ׷ךכ؟ٝفٕ侧״׶㣐ְֹ넝如⯋ד
כⰋ挿׾鸐׷ꟼ侧ָ ׋ֻׁ׿
䒷ֽ׷ַ׵կ鍑׾♧䠐חׅ׷ךחծ#BSUMFUU
ד׮
)BTUJF
ד׮OPSN剑㼭鍑 .1♧菙鷞鍑
׾⟎㹀׃גְ׷ָPWFSQBSBNFUSJ[F 넝如⯋㼗
䕦
׃ג♧菙鷞ד简䕎㔐䌓ׅ׸ל0,הכ䙼ִזְ DG&YUSFNF-FBSOJOH.BDIJOF &-.

3FTFSWPJS$PNQVUJOH
կ׬׃׹չⰋ挿鸐׷ꟼ侧ָ䒷ֽ׍ׯֲպ䚍颵ךקֲָ劤颵涸
⿫罋

View Slide

剣㹱זPWFSGJUUJOHⱄ罋
PolyReg(10)
Train RMSE 0.0189
PolyReg(15)
Train RMSE 0.00737
KernelRidge RBF(γ=1)
Train RMSE 0.103
Train RMSE 0.0139
Train RMSE 0.00726
Train RMSE 0.00726
ֿֿחさ׻ׇ״ֲהׅ׷ה
زٖ٦سؔؿדֿךפ׿ח
מוְ䕦갟ָדג׃תֲ
ؿ؍حذ؍ָؚٝչMPDBMպׄׯזְ
ؿ؍حذ؍ؚٝכչMPDBMպ׌ֽוぐ挿ך鵚⩸כׅץגずׄأ؛٦ٕד䪔׻׸ג׃תֲ

View Slide

寸㹀加ךGJUUJOH 衝侧㔿㹀ד׮PWFSGJUדֹ׷

ExtraTrees (#trees=1, #leaves=8,
bootstrap=off), Train RMSE 0.0189
GBDT w/ 1 ExtraTree (#leaves=8)
Train RMSE 0.29
GBDT w/ 10 ExtraTree (#leaves=8)
Train RMSE 0.17
GBDT w/ 102, ExtraTree (#leaves=8)
Train RMSE 0.00597
GBDT w/ 103, ExtraTree (#leaves=8)
Train RMSE 0.000997
/PUF(#%5X&YUSB5SFFTכTLMFBSOח(#׾
遤׌ֽ剅ֹ䳔ִ׸ל知⽃ח鑐ׇ׷

View Slide

寸㹀加ךGJUUJOH 3'WT&5
3''

Random Forest (#trees=100)
Train RMSE 0.0109
Train RMSE 0.0274
ExtraTrees (#trees=100, bootstrap)
Train RMSE 0.0109
Train RMSE 0.0243
Train RMSE 0.0112
Train RMSE 0.0114
Train RMSE 0.0099
Train RMSE 0.00977
With Random Fourier Features (100-dim Random Kitchen Sinks)

View Slide

L3F-6.-1ךGJUUJOH
1-1-1 ReLU MLP
Train RMSE 0.287
1-5-1 ReLU MLP
Train RMSE 0.287
1-10-1 ReLU MLP
Train RMSE 0.0191
1-100-1 ReLU MLP
Train RMSE 0.00816
Train RMSE 0.0139 Train RMSE 0.0066 Train RMSE 0.0121 Train RMSE 0.0139
˟ⴱ劍⦼ח⣛㶷ׅ׷ל׵אֹכ穠圓֮׷
…
AAAChHichVHLTsJAFD3UF+ID1I2JGyLBuDBkUHzEhSG6cclDHgkS0tYBG0rbtIUEiT+gW40LV5q4MH6AH+DGH3DBJxiXmLhx4aU0MUrE20znzJl77pyZKxmqYtmMtT3C0PDI6Jh33DcxOTXtD8zMZi29bso8I+uqbuYl0eKqovGMrdgqzxsmF2uSynNSda+7n2tw01J07cBuGrxYEyuaUlZk0SYqWS0FQizCnAj2g6gLQnAjoQcecYgj6JBRRw0cGmzCKkRY9BUQBYNBXBEt4kxCirPPcQofaeuUxSlDJLZK/wqtCi6r0bpb03LUMp2i0jBJGUSYvbB71mHP7IG9ss8/a7WcGl0vTZqlnpYbJf/ZfPrjX1WNZhvH36qBnm2UseV4Vci74TDdW8g9fePkqpPeToVbS+yWvZH/G9ZmT3QDrfEu3yV56nqAH4m80ItRg6K/29EPsquR6EYkloyF4rtuq7xYwCKWqR+biGMfCWSoPsc5LnApjAorwpqw3ksVPK5mDj9C2PkCuqOP6w==
k
Optimized With L-BFGS
arch:
(Linear(1, k),
ReLU,
Linear(k, 1))

View Slide

L3F-6.-1ךGJUUJOH
1-1-1 ReLU MLP
Train RMSE 0.289
1-5-1 ReLU MLP
Train RMSE 0.297
1-10-1 ReLU MLP
Train RMSE 0.0237
1-100-1 ReLU MLP
Train RMSE 0.0243
ⴱ劍⦼ח⣛㶷ׅ׷ל׵אֹכ㼰זֻ
ַז׶ך然桦ד؝ٖחז׷
…
AAAChHichVHLTsJAFD3UF+ID1I2JGyLBuDBkUHzEhSG6cclDHgkS0tYBG0rbtIUEiT+gW40LV5q4MH6AH+DGH3DBJxiXmLhx4aU0MUrE20znzJl77pyZKxmqYtmMtT3C0PDI6Jh33DcxOTXtD8zMZi29bso8I+uqbuYl0eKqovGMrdgqzxsmF2uSynNSda+7n2tw01J07cBuGrxYEyuaUlZk0SYqWS0FQizCnAj2g6gLQnAjoQcecYgj6JBRRw0cGmzCKkRY9BUQBYNBXBEt4kxCirPPcQofaeuUxSlDJLZK/wqtCi6r0bpb03LUMp2i0jBJGUSYvbB71mHP7IG9ss8/a7WcGl0vTZqlnpYbJf/ZfPrjX1WNZhvH36qBnm2UseV4Vci74TDdW8g9fePkqpPeToVbS+yWvZH/G9ZmT3QDrfEu3yV56nqAH4m80ItRg6K/29EPsquR6EYkloyF4rtuq7xYwCKWqR+biGMfCWSoPsc5LnApjAorwpqw3ksVPK5mDj9C2PkCuqOP6w==
k
Optimized With Adam
arch:
(Linear(1, k),
ReLU,
Linear(k, 1))

View Slide

LY
3F-6.-1ךGJUUJOH
k=1 ReLU MLP
Train RMSE 0.305
k=5 ReLU MLP
Train RMSE 0.303
k=10 ReLU MLP
Train RMSE 0.0131
k=100 ReLU MLP
Train RMSE 0.00979
Optimized With Adam
arch:
(Linear(1, k),
ReLU,
Linear(k, k),
ReLU,
Linear(k, 1))
x 9 ˟ⴱ劍⦼ח⣛㶷ׅ׷ל׵אֹכ穠圓֮׷
Train RMSE 0.0443 Train RMSE 0.0124 Train RMSE 0.0811 Train RMSE 0.00864

View Slide

הصُ٦ٕٓطحز

View Slide

3F-6/FUXPSL
Daubechies, I., DeVore, R., Foucart, S. et al. Nonlinear Approximation and (Deep) ReLU Networks.
Constr Approx 55, 127–172 (2022). https://doi.org/10.1007/s00365-021-09548-z
Ingrid Daubechies Ronald DeVore
׍ז׫ח׋׌ך،ٕؿ؋كحز갫ծ顑⟣衼罏כ%F7PSF
% ⽃㢌ꆀ
ך㉏겗ד鑫׃ֻ
鍑匿׃גְגⴓַ׶װְׅ

View Slide

3F-6/FUXPSL
3F-6
頾⦼׾縧䳔

View Slide

3F-6/FUXPSL
˖ 3F-6/FUXPSLָ邌植ׅ׷ꟼ侧כչ$POUJOVPVT1JFDFXJTF-JOFBS $1X-
GVODUJPOպ

View Slide

3F-6/FUXPSLWT寸㹀加٥寸㹀啾
GVODUJPOպ
1-1-1 ReLU MLP
Train RMSE 0.289
1-5-1 ReLU MLP
Train RMSE 0.297
1-10-1 ReLU MLP
Train RMSE 0.0237
1-100-1 ReLU MLP
Train RMSE 0.0243
˖ 寸㹀加٥寸㹀啾 װ剑鵚ꦄ岀
ָ邌植ׅ׷ꟼ侧כչ1JFDFXJTF$POTUBOUGVODUJPOպ
Random Forest

View Slide

3F-6/FUXPSL
GVODUJPOպ
1-1-1 ReLU MLP
Train RMSE 0.289
1-5-1 ReLU MLP
Train RMSE 0.297
1-10-1 ReLU MLP
Train RMSE 0.0237
1-100-1 ReLU MLP
Train RMSE 0.0243
Train RMSE 0.0243
&YUSB5SFFT׮⼒ⴓ涸㹀侧זך׌ָ
加侧׾㟓װ׃גְֻהꬊ䌢ח
莆㄂帾ְ䮙⹛׾爙ׅ˘

View Slide

ֿך䚍颵כ&YUSB5SFFTך⯋锷俑ד׮׮׍׹׿陽锷ׁ׸גְ׷
Geurts, P., Ernst, D. & Wehenkel, L. Extremely randomized trees. Mach Learn 63, 3–42 (2006). https://doi.org/10.1007/s10994-006-6226-1

View Slide

ְ׆׸חׇ״걄㚖׀הךMPDBMJUZָ椚鍑ך꒲חז׷
ر٦ة Decision
Tree
Random
Forest GBDT
Nearest
Neighbor
Logistic
Regression
SVM
Gaussian
Process
Neural
Network
1JFDFXJTFDPOTUBOU 1JFDFXJTFMJOFBS

View Slide

3F-6/FUXPSLךⰅ⸂瑞꟦ⴓⶴ׾罋ִ׷
NeurIPS 2019
ICML 2018

View Slide

简䕎أفٓ؎ٝ 㢳㢌ꆀ،ؿ؍ٝأفٓ؎ٝ

"A large class of DNs can be written as a composition of max-aﬃne spline operators (MASOs)"
%JTKPJOUז걄㚖ךQBSUJUJPOָ֮׶
ぐ걄㚖׀החꟼ侧 סאֲ㢳갪䒭
׾
BQQMZׅ׷ֽוչ㞮歲ד鸬竲חז׷
״ֲחպז׏גְ׷ךָأفٓ؎ٝ
׉ך⚥ד׮ⱖ⫷ָչ如㹀侧갪պך
"OFأفٓ؎ٝ׾罋ִ׷
أفٓ؎ٝךؿ؍حذ؍ؚٝכ♧菙חכ
չ걄㚖׀הךⱖ⫷պהչ걄㚖ⴓⶴպךず儗剑黝⻉חז׶ꬊ䌢חꨇ׃ְ

View Slide

.BY"GGJOF4QMJOF ."4

׉ך⚥ד׮ⱖ⫷ָչ如㹀侧갪պך
"OFأفٓ؎ٝ׾罋ִ׷
.BY"OFأفٓ؎ٝדכ僇爙涸ח
չ걄㚖ⴓⶴպ׾罋ִ׷䗳銲ָזְ
%JTKPJOUז걄㚖ךQBSUJUJPOָ֮׶
ぐ걄㚖׀החꟼ侧 סאֲ㢳갪䒭
׾
BQQMZׅ׷ֽוչ㞮歲ד鸬竲חז׷
״ֲחպז׏גְ׷ךָأفٓ؎ٝ

View Slide

.BY"GGJOF4QMJOF ."4

AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmNagEi6AIlnmYByQh7K6TOGRf7E4CMfgDljYWsVGwED/AD7DxByzyCWIZwcbCu5sF0WC8y+ycOXPPnTNzZVNlNiek7xMmJqemZ/yzgbn54EIovLhUsI2WpdC8YqiGVZIlm6pMp3nOuEpLpkUlTVZpUW4eOPvFNrVsZujHvGPSqiY1dFZnisSRKlUMjTakmlgLR0mcuBEZBaIHouBF2gg/QgVOwAAFWqABBR04YhUksPErgwgETOSq0EXOQsTcfQrnEEBtC7MoZkjINvHfwFXZY3VcOzVtV63gKSoOC5URiJEXck8G5Jk8kFfy+WetrlvD8dLBWR5qqVkLXazkPv5VaThzOP1WjfXMoQ67rleG3k2XcW6hDPXts6tBLpmNddfJLXlD/zekT57wBnr7XbnL0GxvjB8ZveCLYYPE3+0YBYXNuLgdT2QS0dS+1yo/rMIabGA/diAFR5CGvNuHS+jBtRAUtoSksDdMFXyeZhl+hHD4BTIfkrs=
!1 AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmY1AJFkERLPMwD0hC2F3HOGRf7E4CMfgDljYWsVGwED/AD7DxByzyCWIZwcbCu5sF0WC8y+ycOXPPnTNzZVNlNiek7xMmJqemZ/yzgbn54EIovLhUtI2WpdCCYqiGVZYlm6pMpwXOuErLpkUlTVZpSW7uO/ulNrVsZuhHvGPSmiY1dHbCFIkjVa4aGm1I9UQ9HCVx4kZkFIgeiIIXGSP8CFU4BgMUaIEGFHTgiFWQwMavAiIQMJGrQRc5CxFz9ymcQwC1LcyimCEh28R/A1cVj9Vx7dS0XbWCp6g4LFRGIEZeyD0ZkGfyQF7J55+1um4Nx0sHZ3mopWY9dLGS//hXpeHM4fRbNdYzhxPYcb0y9G66jHMLZahvn10N8qlcrLtObskb+r8hffKEN9Db78pdluZ6Y/zI6AVfDBsk/m7HKCgm4uJWPJlNRtN7Xqv8sAprsIH92IY0HEIGCm4fLqEH10JQ2BRSwu4wVfB5mmX4EcLBFzQ/krw=
!2 AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmTVAJFkERLPMwD0hC2F3HOGRf7E4CMfgDljYWsVGwED/AD7DxByzyCWIZwcbCu5sF0WC8y+ycOXPPnTNzZVNlNiek7xMmJqemZ/yzgbn54EIovLhUtI2WpdCCYqiGVZYlm6pMpwXOuErLpkUlTVZpSW7uO/ulNrVsZuhHvGPSmiY1dHbCFIkjVa4aGm1I9UQ9HCVx4kZkFIgeiIIXGSP8CFU4BgMUaIEGFHTgiFWQwMavAiIQMJGrQRc5CxFz9ymcQwC1LcyimCEh28R/A1cVj9Vx7dS0XbWCp6g4LFRGIEZeyD0ZkGfyQF7J55+1um4Nx0sHZ3mopWY9dLGS//hXpeHM4fRbNdYzhxPYcb0y9G66jHMLZahvn10N8qlcrLtObskb+r8hffKEN9Db78pdluZ6Y/zI6AVfDBsk/m7HKChuxsWteDKbjKb3vFb5YRXWYAP7sQ1pOIQMFNw+XEIProWgkBBSwu4wVfB5mmX4EcLBFzZfkr0=
!3
˖ (MPCBMMZDPOWFYז⟣䠐ך"OFأفٓ؎ٝ
כ."4ד剅ֽ׷
˖ ."4כ䌢חQJFDFXJTFBOFַאHMPCBMMZ
DPOWFYזךדծו׿זػًٓةד׮䌢ח
DPOUJOVPVT
˖ 鷞ח⟣䠐ךQJFDFXJTFBOF HMPCBMMZDPOWFY
ַאDPOUJOVPVTזꟼ侧כ."4ה׃ג剅ֽ׷կ
AAAChnichVHLSsNAFD2Nr1ofrboR3BRLxVWZarUiCEU3LvuwD6ilJHFaQ9MkJGmhFn9AcGsXrhRciB/gB7jxB1z0E8RlBTcuvE0DosV6w2TOnLnnzpm5kqEqls1Y1yOMjU9MTnmnfTOzc/P+wMJiztIbpsyzsq7qZkESLa4qGs/aiq3ygmFysS6pPC/VDvr7+SY3LUXXjuyWwUt1saopFUUWbaIy6b3NciDEIsyJ4DCIuiAEN5J64BHHOIEOGQ3UwaHBJqxChEVfEVEwGMSV0CbOJKQ4+xzn8JG2QVmcMkRia/Sv0qroshqt+zUtRy3TKSoNk5RBhNkLu2c99swe2Cv7/LNW26nR99KiWRpouVH2XyxnPv5V1Wm2cfqtGunZRgU7jleFvBsO07+FPNA3zzq9zG463F5jt+yN/N+wLnuiG2jNd/kuxdPXI/xI5IVejBoU/d2OYZDbiES3I7FULJTYd1vlxQpWsU79iCOBQySRpfpVXOIKHcErRIQtIT5IFTyuZgk/Qkh8Ab91kFY=
R = 3
AAACjnichVHLSsNAFL2Nr1ofjboR3ARLRRDKREoVQSy66bIP+4BaShKndWheJGmxFn/AvbgQFAUX4gf4AW78ARf9BHFZwY0Lb9KAaLHeMJkzZ+65c2aubKrMdgjpBriR0bHxieBkaGp6ZjbMz80XbKNpKTSvGKphlWTJpirTad5hjkpLpkUlTVZpUW7sufvFFrVsZuj7TtukFU2q66zGFMlBqixVReFYWBPkqljlIyRGvBAGgeiDCPiRNvhHOIBDMECBJmhAQQcHsQoS2PiVQQQCJnIV6CBnIWLePoVTCKG2iVkUMyRkG/iv46rsszqu3Zq2p1bwFBWHhUoBouSF3JMeeSYP5JV8/lmr49VwvbRxlvtaalbDZ4u5j39VGs4OHH2rhnp2oAabnleG3k2PcW+h9PWtk4tebisb7ayQW/KG/m9IlzzhDfTWu3KXodnLIX5k9IIvhg0Sf7djEBTWY2IiFs/EI8ldv1VBWIJlWMV+bEASUpCGvPei53AF1xzPJbhtbqefygV8zQL8CC71BYbukso=
a1x + b1
AAACjnichVHLSsNAFL2Nr1ofjboR3ARLRRDKpJQqglh002Uf9gG1lCSOdWheJGmxFn/AvbgQFAUX4gf4AW78ARf9BHFZwY0Lb9KAaLHeMJkzZ+65c2aubKrMdgjpBriR0bHxieBkaGp6ZjbMz80XbaNpKbSgGKphlWXJpirTacFhjkrLpkUlTVZpSW7suvulFrVsZuh7TtukVU2q6+yQKZKDVEWqxYVjYU2Qa/EaHyEx4oUwCEQfRMCPjME/wj4cgAEKNEEDCjo4iFWQwMavAiIQMJGrQgc5CxHz9imcQgi1TcyimCEh28B/HVcVn9Vx7da0PbWCp6g4LFQKECUv5J70yDN5IK/k889aHa+G66WNs9zXUrMWPlvMf/yr0nB24OhbNdSzA4ew4Xll6N30GPcWSl/fOrno5Tdz0c4KuSVv6P+GdMkT3kBvvSt3WZq7HOJHRi/4Ytgg8Xc7BkExHhOTsUQ2EUnt+K0KwhIswyr2Yx1SkIYMFLwXPYcruOZ4Lsltcdv9VC7gaxbgR3DpL4s2ksw=
a2x + b2
AAACjnichVHLSsNAFL2Nr1ofjboR3ARLRRDKxJYqglh002Uf9gG1hCROa2heJGmxFn/AvbgQFAUX4gf4AW78ARf9BHFZwY0Lb9KAaLHeMJkzZ+65c2auZKqK7RDSDTAjo2PjE8HJ0NT0zGyYnZsv2kbTkmlBNlTDKkuiTVVFpwVHcVRaNi0qapJKS1Jjz90vtahlK4a+77RNWtXEuq7UFFl0kKqIQpw75tY4SYgLbITEiBfcIOB9EAE/Mgb7CAdwCAbI0AQNKOjgIFZBBBu/CvBAwESuCh3kLESKt0/hFEKobWIWxQwR2Qb+67iq+KyOa7em7allPEXFYaGSgyh5IfekR57JA3kln3/W6ng1XC9tnKW+lppC+Gwx//GvSsPZgaNv1VDPDtRg0/OqoHfTY9xbyH196+Sil9/KRTsr5Ja8of8b0iVPeAO99S7fZWnucogfCb3gi2GD+N/tGATF9RifjCWyiUhq129VEJZgGVaxHxuQgjRkoOC96DlcwTXDMklmm9nppzIBX7MAP4JJfwGPfpLO
a3x + b3
.BY"OFأفٓ؎ٝדכ僇爙涸ח
չ걄㚖ⴓⶴպ׾罋ִ׷䗳銲ָזְ

View Slide

.BY"GGJOF4QMJOF0QFSBUPS ."40

AAAChHichVG7SgNBFD1ZNcZ31EawCQbFQsJE4wMLCdoINiYxUYgiu+skDm52l91JIAZ/QFvFwkrBQvwAP8DGH7DIJ4hlBBsLbzYLokG9y+ycOXPPnTNzNdsQrmSsHlA6OruC3aGe3r7+gcGh8PBIzrXKjs6zumVYzo6mutwQJs9KIQ2+YztcLWkG39aO1pr72xXuuMIyt2TV5nsltWiKgtBVSVRqYz8cZTHmRaQdxH0QhR+bVvgBuziABR1llMBhQhI2oMKlL484GGzi9lAjziEkvH2OE/SStkxZnDJUYo/oX6RV3mdNWjdrup5ap1MMGg4pI5hkz+yONdgTu2cv7OPXWjWvRtNLlWatpeX2/tDpWOb9X1WJZonDL9WfniUKWPK8CvJue0zzFnpLXzm+bGSW05O1KXbDXsn/NauzR7qBWXnTb1M8ffWHH4280ItRg+I/29EOcrOx+EIskUpEk6t+q0IYxwSmqR+LSGIdm8hSfY4znONCCSozypwy30pVAr5mFN9CWfkEdqOPyw==
K
AAACnHichVG7SgNBFL2urxgfiVooCBIMEaswkaBiJZpCEMEk5gGJhp11kgzZF7uToC75AX/AwkrFQtTaD7DxByzyCWIZwcbCm82CqKh3mZ0zZ+65c2YuNVVuC0JaPVJvX//AoG/IPzwyOhYIjk9kbaNuKSyjGKph5alsM5XrLCO4UFnetJisUZXlaG2js59rMMvmhr4rjky2p8kVnZe5IgukSsGpItWcw2aoyPVQUZNFlVIn1dxPlIJhEiVuhH6CmAfC4MWOEbyHIhyAAQrUQQMGOgjEKshg41eAGBAwkdsDBzkLEXf3GTTBj9o6ZjHMkJGt4b+Cq4LH6rju1LRdtYKnqDgsVIYgQp7INWmTR3JDnsn7r7Uct0bHyxHOtKtlZilwMp1++1el4Syg+qn607OAMqy4Xjl6N12mcwulq28cn7bTq6mIM08uyAv6Pyct8oA30BuvylWSpc7+8EPRC74YNij2vR0/QXYxGluKxpPx8Nq61yofzMAcLGA/lmENNmEHMq6vS7iFO2lWSkhb0nY3VerxNJPwJaTsB6B/mVE=
x 2 RD
."40
."40
,⦐ך."4׾⢪׏׋
0QFSBUPS

View Slide

3F-6/FUXPSL."40ךさ䧭ꟼ侧
AAAC7nichVFNa9RQFD1NrdapOmPdCG6CQ2VmM7yUYotQKLpx4aJf0xaaNiSvLzOP5ovkzdAa8gf8AboQEQVB8Qf4A9y4c+WiP6F02YqbUr3JpJRarDck79zz7rk5710n8mSiGNsb0oavjFy9Nnq9Mnbj5q1q7fb4ShL2Yi7aPPTCeM2xE+HJQLSVVJ5Yi2Jh+44nVp3tJ/n+al/EiQyDZbUbiQ3f7gTSldxWRFk16Vrmclcou2E6frqTNfVZ3fSEqxoVdzNtPGtmVmqqvGCQZbrJZcx1cytUSYnzQuNcIWVZxYxlp6uap42tWp21WBH6RWCUoI4y5sPaF5jYQgiOHnwIBFCEPdhI6FmHAYaIuA2kxMWEZLEvkKFC2h5VCaqwid2mb4ey9ZINKM97JoWa0188emNS6phgP9gndsi+sc9snx3/s1da9Mi97NLqDLQisqov7i79+q/Kp1Whe6a61LOCi5nCqyTvUcHkp+ADff/5q8OlR4sT6QP2nh2Q/3dsj32lEwT9I/5hQSy+vsSPQ17oxmhAxt/juAhWJlvGw9bUwlR97nE5qlHcw300aB7TmMNTzKNN/b/jJ07wW4u0l9ob7e2gVBsqNXdwLrSPfwCSMriX
f⇥(x) =
⇣
f(L)
✓(L)
· · · f(1)
✓(1)
⌘
(x) AAACvHicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuYx4RkpJYkKtgqxOSkppXEVMeUgPhx1RqGmrU6MSn5JcU6cCEfzdqYosz0jJKY2ngBZQM9AzBQwGQYQhnKDFAQkC+wnSGGIYUhnyGZoZQhlyGVIY+hBMjOYUhkKAbCaAZDBgOGAqBYLEM1UKwIyMoEy6cy1DJwAfWWAlWlAlUkAkWzgWQ6kBcNFc0D8kFmFoN1JwNtyQHiIqBOBQZVg6sGKw0+G5wwWG3w0uAPTrOqwWaA3FIJpJMgelML4vm7JIK/E9SVC6RLGDIQuvC6uYQhjcEC7NZMoNsLwCIgXyRD9JdVTf8cbBWkWq1msMjgNdD9Cw1uGhwG+iCv7Evy0sDUoNl43JMEdAswxIARZIgeHZiMMCM9QzM9k0ATZQcnaFRxMEgzKDFoAOPDnMGBwYMhgCEUaP5chsMMZxjOMtkxpTBlM+VClDIxQvUIM6AApjIALx+l6g==
⇥ =
n
✓(1), . . . , ✓(L)
o
AAACm3ichVHLSsNAFD2N73dVEEEEsVTqpkxFVFyJbkRc9GFVsLUkcaqDaRKSaUFjf8AfcOFKwYXo3g9w4w+46CeISwU3LrxNA6LFekNyz5y55+bMXM02hCsZq4WUtvaOzq7unt6+/oHBofDwyLZrlR2dZ3XLsJxdTXW5IUyelUIafNd2uFrSDL6jHa/V93cq3HGFZW7JE5vnS+qhKYpCVyVRhfBYseDl5BGX6r4XE7PVaiMVwhEWZ35MNYNEACIIImmFH5DDASzoKKMEDhOSsAEVLj17SIDBJi4PjziHkPD3OaroJW2ZqjhVqMQe0/eQVnsBa9K63tP11Tr9xaDXIeUUouyZ3bI39sTu2Av7/LOX5/eoezmhrDW03C4MnY9nPv5VlShLHH2rWnqWKGLJ9yrIu+0z9VPoDX3l9OIts5yOejPsmr2S/ytWY490ArPyrt+kePqyhR+NvNCN0YASv8fRDLbn4omF+HxqPrKyGoyqGxOYRozmsYgVrCOJLPU/wzXucK9MKmvKhrLZKFVCgWYUP0LJfgF7g5jS
f(i)
✓(i)
• fully-connected
• convolution
• activation
• ReLU
• leaky ReLU
• absolute value
• pooling (max, average, channel, etc)
• recurrent
• skip connection
MASO
DNs are signal-dependent aﬃne
transformations. The particular aﬃne
mapping applied to x depends on which
parrtition of the spline it falls in Rd.

View Slide

3F-6/FUXPSLךⰅ⸂瑞꟦ⴓⶴ
ReLU
x1
x2
y
x1
x2
AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTg2JcEd245CGPBAlp64gNpW3aQgLEHzBxKwtXmrgwfoAf4MYfcMEnGJeYuHHhpTQxSsTbTOfMmXvunJkrm5pqO4z1fMLE5NT0jH82MDe/sBgMLS3nbaNhKTynGJphFWXJ5pqq85yjOhovmhaX6rLGC3LtYLBfaHLLVg39yGmZvFyXqrp6qiqSQ1S2XREroQiLMTfCo0D0QARepIzQI45xAgMKGqiDQ4dDWIMEm74SRDCYxJXRIc4ipLr7HOcIkLZBWZwyJGJr9K/SquSxOq0HNW1XrdApGg2LlGFE2Qu7Z332zB7YK/v8s1bHrTHw0qJZHmq5WQlerGY//lXVaXZw9q0a69nBKXZdryp5N11mcAtlqG+2u/3sXiba2WC37I3837Aee6Ib6M135S7NM9dj/MjkhV6MGiT+bscoyG/FxJ1YPB2PJPe9VvmxhnVsUj8SSOIQKeSofhWXuEJX8AsxYVtIDFMFn6dZwY8Qkl9Y9pCe
z1
AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDQTGuiG5c8pBHgoS0dcCG0jZtIQHiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVDFWxbMb6HmFqemZ2zjvvW1hcWvYHVlbzlt40ZZ6TdVU3i5JocVXReM5WbJUXDZOLDUnlBal+ONwvtLhpKbp2bLcNXm6INU2pKrJoE5XtVGKVQIhFmBPBcRB1QQhupPTAI05wCh0ymmiAQ4NNWIUIi74SomAwiCujS5xJSHH2Oc7hI22TsjhliMTW6V+jVcllNVoPa1qOWqZTVBomKYMIsxd2zwbsmT2wV/b5Z62uU2PopU2zNNJyo+K/WM9+/Ktq0Gzj7Fs10bONKvYcrwp5NxxmeAt5pG91eoPsfibc3WK37I3837A+e6IbaK13+S7NM9cT/EjkhV6MGhT93Y5xkI9ForuReDoeSh64rfJiA5vYpn4kkMQRUshR/RoucYWe4BUiwo6QGKUKHlezhh8hJL8AWxaQnw==
z2
AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCKDohhXRDcuecgjQULaOmBDaZu2kADxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmM9jzAxOTU94531zc0vLPoDS8s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFQ7HOznm9y0FF07tlsGL9XFqqZUFFm0icq0y9vlQIhFmBPBURB1QQhuJPXAI05wCh0yGqiDQ4NNWIUIi74iomAwiCuhQ5xJSHH2Oc7hI22DsjhliMTW6F+lVdFlNVoPalqOWqZTVBomKYMIsxd2z/rsmT2wV/b5Z62OU2PgpUWzNNRyo+y/WM18/Kuq02zj7Fs11rONCvYcrwp5NxxmcAt5qG+2u/3Mfjrc2WC37I3837Aee6IbaM13+S7F09dj/EjkhV6MGhT93Y5RkNuKRHcjsVQslDhwW+XFGtaxSf2II4EjJJGl+lVc4gpdwStEhB0hPkwVPK5mBT9CSHwBXTaQoA==
z3
AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhgUIwV0caShzwSJGR3HXHDvrK7kADxB0xspbDSxML4AX6AjT9gwScYS0xsLLwsmxgl4t3Mzpkz99w5M1cyVcV2GOv5hInJqekZ/2xgbn5hMRhaWi7YRsOSeV42VMMqSaLNVUXneUdxVF4yLS5qksqLUv1gsF9scstWDP3IaZm8ook1XTlVZNEhKteuJqqhCIsxN8KjIO6BCLxIG6FHHOMEBmQ0oIFDh0NYhQibvjLiYDCJq6BDnEVIcfc5zhEgbYOyOGWIxNbpX6NV2WN1Wg9q2q5aplNUGhYpw4iyF3bP+uyZPbBX9vlnrY5bY+ClRbM01HKzGrxYzX38q9JodnD2rRrr2cEpdl2vCnk3XWZwC3mob7a7/dxeNtrZYLfsjfzfsB57ohvozXf5LsOz12P8SOSFXowaFP/djlFQ2IrFd2KJTCKS2vda5cca1rFJ/UgihUOkkaf6NVziCl3BL8SEbSE5TBV8nmYFP0JIfQFfVpCh
z4
AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCKDATGuiG5c8pBHgoS0dcCG0jZtIQHiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVDFWxbMb6HmFqemZ2zjvvW1hcWvYHVlbzlt40ZZ6TdVU3i5JocVXReM5WbJUXDZOLDUnlBal+ONwvtLhpKbp2bLcNXm6INU2pKrJoE5XtVOKVQIhFmBPBcRB1QQhupPTAI05wCh0ymmiAQ4NNWIUIi74SomAwiCujS5xJSHH2Oc7hI22TsjhliMTW6V+jVcllNVoPa1qOWqZTVBomKYMIsxd2zwbsmT2wV/b5Z62uU2PopU2zNNJyo+K/WM9+/Ktq0Gzj7Fs10bONKvYcrwp5NxxmeAt5pG91eoPsfibc3WK37I3837A+e6IbaK13+S7NM9cT/EjkhV6MGhT93Y5xkN+JRHcjsXQslDxwW+XFBjaxTf1IIIkjpJCj+jVc4go9wStEhLiQGKUKHlezhh8hJL8AYXaQog==
z5
z1 AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDQTGuiG5c8pBHgoS0dcCG0jZtIQHiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVDFWxbMb6HmFqemZ2zjvvW1hcWvYHVlbzlt40ZZ6TdVU3i5JocVXReM5WbJUXDZOLDUnlBal+ONwvtLhpKbp2bLcNXm6INU2pKrJoE5XtVGKVQIhFmBPBcRB1QQhupPTAI05wCh0ymmiAQ4NNWIUIi74SomAwiCujS5xJSHH2Oc7hI22TsjhliMTW6V+jVcllNVoPa1qOWqZTVBomKYMIsxd2zwbsmT2wV/b5Z62uU2PopU2zNNJyo+K/WM9+/Ktq0Gzj7Fs10bONKvYcrwp5NxxmeAt5pG91eoPsfibc3WK37I3837A+e6IbaK13+S7NM9cT/EjkhV6MGhT93Y5xkI9ForuReDoeSh64rfJiA5vYpn4kkMQRUshR/RoucYWe4BUiwo6QGKUKHlezhh8hJL8AWxaQnw==
z2 AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCKDohhXRDcuecgjQULaOmBDaZu2kADxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmM9jzAxOTU94531zc0vLPoDS8s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFQ7HOznm9y0FF07tlsGL9XFqqZUFFm0icq0y9vlQIhFmBPBURB1QQhuJPXAI05wCh0yGqiDQ4NNWIUIi74iomAwiCuhQ5xJSHH2Oc7hI22DsjhliMTW6F+lVdFlNVoPalqOWqZTVBomKYMIsxd2z/rsmT2wV/b5Z62OU2PgpUWzNNRyo+y/WM18/Kuq02zj7Fs11rONCvYcrwp5NxxmcAt5qG+2u/3Mfjrc2WC37I3837Aee6IbaM13+S7F09dj/EjkhV6MGhT93Y5RkNuKRHcjsVQslDhwW+XFGtaxSf2II4EjJJGl+lVc4gpdwStEhB0hPkwVPK5mBT9CSHwBXTaQoA==
z3 AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhgUIwV0caShzwSJGR3HXHDvrK7kADxB0xspbDSxML4AX6AjT9gwScYS0xsLLwsmxgl4t3Mzpkz99w5M1cyVcV2GOv5hInJqekZ/2xgbn5hMRhaWi7YRsOSeV42VMMqSaLNVUXneUdxVF4yLS5qksqLUv1gsF9scstWDP3IaZm8ook1XTlVZNEhKteuJqqhCIsxN8KjIO6BCLxIG6FHHOMEBmQ0oIFDh0NYhQibvjLiYDCJq6BDnEVIcfc5zhEgbYOyOGWIxNbpX6NV2WN1Wg9q2q5aplNUGhYpw4iyF3bP+uyZPbBX9vlnrY5bY+ClRbM01HKzGrxYzX38q9JodnD2rRrr2cEpdl2vCnk3XWZwC3mob7a7/dxeNtrZYLfsjfzfsB57ohvozXf5LsOz12P8SOSFXowaFP/djlFQ2IrFd2KJTCKS2vda5cca1rFJ/UgihUOkkaf6NVziCl3BL8SEbSE5TBV8nmYFP0JIfQFfVpCh
z4 AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCKDATGuiG5c8pBHgoS0dcCG0jZtIQHiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVDFWxbMb6HmFqemZ2zjvvW1hcWvYHVlbzlt40ZZ6TdVU3i5JocVXReM5WbJUXDZOLDUnlBal+ONwvtLhpKbp2bLcNXm6INU2pKrJoE5XtVOKVQIhFmBPBcRB1QQhupPTAI05wCh0ymmiAQ4NNWIUIi74SomAwiCujS5xJSHH2Oc7hI22TsjhliMTW6V+jVcllNVoPa1qOWqZTVBomKYMIsxd2zwbsmT2wV/b5Z62uU2PopU2zNNJyo+K/WM9+/Ktq0Gzj7Fs10bONKvYcrwp5NxxmeAt5pG91eoPsfibc3WK37I3837A+e6IbaK13+S7NM9cT/EjkhV6MGhT93Y5xkN+JRHcjsXQslDxwW+XFBjaxTf1IIIkjpJCj+jVc4go9wStEhLiQGKUKHlezhh8hJL8AYXaQog==
z5
ぐ걄㚖דכ♴鎸׾
穈׫さ׻ׇ׋
"OF㢌䳔ָ黝欽
ׁ׸׷
㞮歲ד鸬竲חז׷

View Slide

z1 AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDQTGuiG5c8pBHgoS0dcCG0jZtIQHiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVDFWxbMb6HmFqemZ2zjvvW1hcWvYHVlbzlt40ZZ6TdVU3i5JocVXReM5WbJUXDZOLDUnlBal+ONwvtLhpKbp2bLcNXm6INU2pKrJoE5XtVGKVQIhFmBPBcRB1QQhupPTAI05wCh0ymmiAQ4NNWIUIi74SomAwiCujS5xJSHH2Oc7hI22TsjhliMTW6V+jVcllNVoPa1qOWqZTVBomKYMIsxd2zwbsmT2wV/b5Z62uU2PopU2zNNJyo+K/WM9+/Ktq0Gzj7Fs10bONKvYcrwp5NxxmeAt5pG91eoPsfibc3WK37I3837A+e6IbaK13+S7NM9cT/EjkhV6MGhT93Y5xkI9ForuReDoeSh64rfJiA5vYpn4kkMQRUshR/RoucYWe4BUiwo6QGKUKHlezhh8hJL8AWxaQnw==
z2 AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCKDohhXRDcuecgjQULaOmBDaZu2kADxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmM9jzAxOTU94531zc0vLPoDS8s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFQ7HOznm9y0FF07tlsGL9XFqqZUFFm0icq0y9vlQIhFmBPBURB1QQhuJPXAI05wCh0yGqiDQ4NNWIUIi74iomAwiCuhQ5xJSHH2Oc7hI22DsjhliMTW6F+lVdFlNVoPalqOWqZTVBomKYMIsxd2z/rsmT2wV/b5Z62OU2PgpUWzNNRyo+y/WM18/Kuq02zj7Fs11rONCvYcrwp5NxxmcAt5qG+2u/3Mfjrc2WC37I3837Aee6IbaM13+S7F09dj/EjkhV6MGhT93Y5RkNuKRHcjsVQslDhwW+XFGtaxSf2II4EjJJGl+lVc4gpdwStEhB0hPkwVPK5mBT9CSHwBXTaQoA==
z3 AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhgUIwV0caShzwSJGR3HXHDvrK7kADxB0xspbDSxML4AX6AjT9gwScYS0xsLLwsmxgl4t3Mzpkz99w5M1cyVcV2GOv5hInJqekZ/2xgbn5hMRhaWi7YRsOSeV42VMMqSaLNVUXneUdxVF4yLS5qksqLUv1gsF9scstWDP3IaZm8ook1XTlVZNEhKteuJqqhCIsxN8KjIO6BCLxIG6FHHOMEBmQ0oIFDh0NYhQibvjLiYDCJq6BDnEVIcfc5zhEgbYOyOGWIxNbpX6NV2WN1Wg9q2q5aplNUGhYpw4iyF3bP+uyZPbBX9vlnrY5bY+ClRbM01HKzGrxYzX38q9JodnD2rRrr2cEpdl2vCnk3XWZwC3mob7a7/dxeNtrZYLfsjfzfsB57ohvozXf5LsOz12P8SOSFXowaFP/djlFQ2IrFd2KJTCKS2vda5cca1rFJ/UgihUOkkaf6NVziCl3BL8SEbSE5TBV8nmYFP0JIfQFfVpCh
z4 AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCKDATGuiG5c8pBHgoS0dcCG0jZtIQHiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVDFWxbMb6HmFqemZ2zjvvW1hcWvYHVlbzlt40ZZ6TdVU3i5JocVXReM5WbJUXDZOLDUnlBal+ONwvtLhpKbp2bLcNXm6INU2pKrJoE5XtVOKVQIhFmBPBcRB1QQhupPTAI05wCh0ymmiAQ4NNWIUIi74SomAwiCujS5xJSHH2Oc7hI22TsjhliMTW6V+jVcllNVoPa1qOWqZTVBomKYMIsxd2zwbsmT2wV/b5Z62uU2PopU2zNNJyo+K/WM9+/Ktq0Gzj7Fs10bONKvYcrwp5NxxmeAt5pG91eoPsfibc3WK37I3837A+e6IbaK13+S7NM9cT/EjkhV6MGhT93Y5xkN+JRHcjsXQslDxwW+XFBjaxTf1IIIkjpJCj+jVc4go9wStEhLiQGKUKHlezhh8hJL8AYXaQog==
z5
-12.43 15.48 2.04 2.05 -2.48

View Slide

arch:
(Linear(2, 5),
ReLU,
Linear(5, 1))
ReLU
x1
x2
y
x1
x2
x1
x2
x1
x2
x1
x2

View Slide

Power diagram (PD)
aka Laguerre–Voronoi diagram

View Slide

㢳㾴חז׷ꥷחכ걄㚖׀הח殯ז׷걄㚖ⴓⶴד稢ⴓׁ׸גְֻ

View Slide

Balestriero, Randall. "Max-Aﬃne Splines Insights Into Deep Learning." (2021) Diss., Rice University. https://hdl.handle.net/1911/110439.
1JFDFXJTF-JOFBSדչ4QMJOFպ 㞮歲ד鸬竲
הְֲֿהכ馄䎂꬗׾ؙءٍؙءٍח׃׋朐䡾

View Slide

/FVSBMOFUXPSLBTMPDBMJUZTFOTJUJWFIBTIJOH

View Slide

寸㹀加ךⰅ⸂瑞꟦ⴓⶴ

View Slide

✓1
X2
 ✓1
yes no

View Slide

✓1
X2
 ✓1
yes no
✓2
X1
 ✓2
yes no
Blue

View Slide

✓1
X2
 ✓1
yes no
✓2
X1
 ✓2
yes no
Blue
✓3
AAAClnichVHLSsNAFD3GV3221Y3gplgUV2WiRcWFFEV0Wa3VgpWQxKmGpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxKWCGxfepAFRUW+YzJkz99w5M1dzTMMTjDVapNa29o7OSFd3T29ffzQWH1j37Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tPKCv79R465n2NaaOHD4VkXdsYySoauCKCUWLyhyomjyvURR7HKhKpNKLMlSLIjETyCHIIkwsnbsDkVsw4aOKirgsCAIm1Dh0bcJGQwOcVuoE+cSMoJ9jiN0k7ZKWZwyVGLL9N+h1WbIWrT2a3qBWqdTTBouKRMYZY/shr2wB3bLntj7r7XqQQ3fywHNWlPLHSV6PJR7+1dVoVlg91P1p2eBEmYCrwZ5dwLGv4Xe1NcOT15ys6uj9TF2wZ7J/zlrsHu6gVV71S9X+OrpH3408kIvRg2Sv7fjJ1ifSMlTqfRKOpmZD1sVwTBGME79mEYGy8giT/X3cYYrXEtD0py0KC01U6WWUDOILyFlPwA1vpYb
X1
 ✓3
yes no
Red

View Slide

✓1
X2
 ✓1
yes no
✓2
X1
 ✓2
yes no
Blue
✓4
X2
 ✓4
yes no
Red
✓3
X1
 ✓3
yes no
Red

View Slide

✓1
X2
 ✓1
yes no
✓2
X1
 ✓2
yes no
Blue
✓4
X2
 ✓4
yes no
Red
✓5
AAAClnichVHLSsNAFD3GV62vVjeCm2JRXJWJ+MKFFEV0WVurBSshiVMNTZOYTCta/AF/wIW4UFARP8APcOMPuOgniEsFNy68SQOiot4wmTNn7rlzZq7mmIYnGGu0SK1t7R2dka5od09vX38sPrDu2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvaOVFf3+jxl3PsK01ceDwrYq6YxklQ1cFUUosXlDkRNHke4mi2OVCVaaUWJKlWBCJn0AOQRJhZOzYHYrYhg0dVVTAYUEQNqHCo28TMhgc4rZQJ84lZAT7HEeIkrZKWZwyVGLL9N+h1WbIWrT2a3qBWqdTTBouKRMYZY/shr2wB3bLntj7r7XqQQ3fywHNWlPLHaX/eCj39q+qQrPA7qfqT88CJcwGXg3y7gSMfwu9qa8dnrzk5rKj9TF2wZ7J/zlrsHu6gVV71S9Xefb0Dz8aeaEXowbJ39vxE6xPpOTp1OTqZDK9ELYqgmGMYJz6MYM0VpBBnurv4wxXuJaGpHlpSVpupkotoWYQX0LKfAA5/pYd
X1
 ✓5
yes no
Red
✓3
X1
 ✓3
yes no
Red

View Slide

✓1
X2
 ✓1
yes no
✓2
X1
 ✓2
yes no
Blue
✓4
X2
 ✓4
yes no
Red
✓5
X1
 ✓5
yes no
Red
✓6
AAAClnichVHLSsNAFD3Gd321uhHcFIviqkykVHEhRRFd9mG1YCUkcWyDaRKTaUWLP+APuBAXCiriB/gBbvwBF/0EcVnBjQtv04CoqDdM5syZe+6cmas5puEJxhodUmdXd09vX39oYHBoeCQcGd3w7Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tb3l1v5mjbueYVvr4tDh2xW1ZBm7hq4KopRwpKDI0aLJ96NFUeZCVZJKOMbizI/oTyAHIIYg0nb4HkXswIaOKirgsCAIm1Dh0bcFGQwOcduoE+cSMvx9jmOESFulLE4ZKrF79C/RaitgLVq3anq+WqdTTBouKaOYYk/sljXZI7tjz+z911p1v0bLyyHNWlvLHWXkZDz39q+qQrNA+VP1p2eBXcz7Xg3y7vhM6xZ6W187Om3mFrJT9Wl2yV7I/wVrsAe6gVV71a8yPHv2hx+NvNCLUYPk7+34CTZm43IynsgkYqmloFV9mMAkZqgfc0hhDWnkqf4BznGNG2lcWpRWpNV2qtQRaMbwJaT0Bzwelh4=
X1
 ✓6
yes no
Red
Blue
✓3
X1
 ✓3
yes no
Red

View Slide

✓1
X2
 ✓1
yes no
✓2
X1
 ✓2
yes no
Blue
✓4
X2
 ✓4
yes no
Red
✓5
X1
 ✓5
yes no
Red
✓6
AAAClnichVHLSsNAFD3Gd321uhHcFIviqkykVHEhRRFd9mG1YCUkcWyDaRKTaUWLP+APuBAXCiriB/gBbvwBF/0EcVnBjQtv04CoqDdM5syZe+6cmas5puEJxhodUmdXd09vX39oYHBoeCQcGd3w7Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tb3l1v5mjbueYVvr4tDh2xW1ZBm7hq4KopRwpKDI0aLJ96NFUeZCVZJKOMbizI/oTyAHIIYg0nb4HkXswIaOKirgsCAIm1Dh0bcFGQwOcduoE+cSMvx9jmOESFulLE4ZKrF79C/RaitgLVq3anq+WqdTTBouKaOYYk/sljXZI7tjz+z911p1v0bLyyHNWlvLHWXkZDz39q+qQrNA+VP1p2eBXcz7Xg3y7vhM6xZ6W187Om3mFrJT9Wl2yV7I/wVrsAe6gVV71a8yPHv2hx+NvNCLUYPk7+34CTZm43IynsgkYqmloFV9mMAkZqgfc0hhDWnkqf4BznGNG2lcWpRWpNV2qtQRaMbwJaT0Bzwelh4=
X1
 ✓6
yes no
Red
Blue
✓3
X1
 ✓3
yes no
Red
AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRQIoZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYS9SCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4XtaGw3Gs/Ew6kDt1VeWIcN2MJ+JCAFx5CGvNOHS+jDteAXdoSksD9KFTyuZhV+hHD0BVqrks4=
✓7
AAAClnichVHLSsNAFD3Gd321uhHcFIviqkykWHEhRRFd9mG1YCUkcWyDaRKTaUWLP+APuBAXCiriB/gBbvwBF/0EcVnBjQtv04CoqDdM5syZe+6cmas5puEJxhodUmdXd09vX39oYHBoeCQcGd3w7Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tb3l1v5mjbueYVvr4tDh2xW1ZBm7hq4KopRwpKDI0aLJ96NFUeZCVZJKOMbizI/oTyAHIIYg0nb4HkXswIaOKirgsCAIm1Dh0bcFGQwOcduoE+cSMvx9jmOESFulLE4ZKrF79C/RaitgLVq3anq+WqdTTBouKaOYYk/sljXZI7tjz+z911p1v0bLyyHNWlvLHWXkZDz39q+qQrNA+VP1p2eBXcz7Xg3y7vhM6xZ6W187Om3mFrJT9Wl2yV7I/wVrsAe6gVV71a8yPHv2hx+NvNCLUYPk7+34CTZm4/JcPJFJxFJLQav6MIFJzFA/kkhhDWnkqf4BznGNG2lcWpRWpNV2qtQRaMbwJaT0Bz4+lh8=
X1
 ✓7
yes no
Blue Blue

View Slide

寸㹀加כ걄㚖ⴓⶴ♳ך㢳如⯋ؼأزؚٓي ⼒ⴓ涸㹀侧✮庠

19
8
35 2
predict_proba(x)
red blue
8/27 = 0.296
19/27 = 0.704
0.0
1.0
0.704
0.296
ⴓ겲加
Piecewise "constant"
̔չ(SFFEZד㧅䔲䚍׮ֶראַזְذؗز٦ז걄㚖ⴓⶴպחչ馄؝ٝ؟غז✮庠պ׾䫴さׇ
̔窟鎘涸חכקה׿ו穗꿀ⴓ䋒ח鵚ְ葺ְ䚍颵׾䭯א 䝤ְֿהָꬊ䌢ח饯ֹב׵ְ

View Slide

㔐䌓加


View Slide

׉ך걄㚖חֶ׍׋
؟ٝفٕך䎂㖱⦼
㔐䌓加


View Slide

寸㹀加،ٝ؟ٝـٕכ׉ך ꅾ׫➰ֹ
ㄤ
= + +
+ + +
RandomForestClassiﬁer
(n_estimators=6,
max_leaf_nodes=4)
6 × DecisionTreeClassiﬁer(max_leaf_nodes=4)
،ٝ؟ٝـٕ䖓ך
걄㚖侧כ穈さׇד
״׶㣐䌴ח㟓ִ׷
⸇岀ٌرٕ ㄤ

ח״׷걄㚖ך稢ⴓ
4QMJOFה殯ז׶
㞮歲דך鸬竲䚍כ
Ⰻֻ䬐⥂ׁ׸זְ
׋׌׃،ٝ؟ٝـٕ
ח״׷䎂徽⻉⸬卓ד
׭׍ׯ׭׍ׯ♶鸬竲
חכז׶ב׵ְ

View Slide

寸㹀加،ٝ؟ٝـٕכ׉ך ꅾ׫➰ֹ
ㄤ
RandomForestRegressor
(n_estimators=6,
max_leaf_nodes=8)
= + +
+ + +
6 × DecisionTreeRegressor(max_leaf_nodes=8)
4QMJOFה殯ז׶
Ⰻֻ䬐⥂ׁ׸זְ
׋׌׃،ٝ؟ٝـٕ
חכז׶ב׵ְ
⸇岀ٌرٕ ㄤ


View Slide

Random forests ntree=10
Extra trees ntree=10
4QMJOFה殯ז׶
Ⰻֻ䬐⥂ׁ׸זְ
׋׌׃،ٝ؟ٝـٕ
חכז׶ב׵ְ
⸇岀ٌرٕ ㄤ


View Slide

⿫罋0CMJWJPVT5SFFTךⴓⶴ
寸㹀加׾ع٦س⻉ׅ׷הַ4PGU5SFFTזו鎘皾ָطحؙחז׷㜥さծ
ずٖׄكٕךTQMJUUFSךⰟ剣 0CMJWJPVT5SFFT
ָ剣⸬ /0%& $BU#PPTU FUD

0CMJWJPVTח׃״ֲָ׃תְָծ׮׃加侧ָꬊ䌢ח㣐ֹֻדֹ׷ז׵וֲׇקר
瑞꟦ًحءُ涸圓鸡חז׷ךד鵚⡂腉⸂♳קר䊴כזֻծ鎘皾⸬桦⻉ך䛷䜋׌ֽ
׾ֲתֻ❦「דֹ׷ךַ׮
✓1
✓2
✓3
AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTg2JcEd245CGPBAlp64ANpW3aQoLEHzBxKwtXmrgwfoAf4MYfcMEnGJeYuHHhpTQxSsTbTOfMmXvunJkrm5pqO4z1fMLE5NT0jH82MDe/sBgMLS3nbaNpKTynGJphFWXJ5pqq85yjOhovmhaXGrLGC3L9YLBfaHHLVg39yGmbvNyQarpaVRXJISpbrIiVUITFmBvhUSB6IAIvUkboEcc4gQEFTTTAocMhrEGCTV8JIhhM4sroEGcRUt19jnMESNukLE4ZErF1+tdoVfJYndaDmrarVugUjYZFyjCi7IXdsz57Zg/slX3+Wavj1hh4adMsD7XcrAQvVrMf/6oaNDs4/VaN9eygil3Xq0reTZcZ3EIZ6ltn3X52LxPtbLBb9kb+b1iPPdEN9Na7cpfmmesxfmTyQi9GDRJ/t2MU5Ldi4k4sno5Hkvteq/xYwzo2qR8JJHGIFHJUv4ZLXKEr+IWYsC0khqmCz9Os4EcIyS8QcpB8
X1
AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhAUIwV0caShzwSJGR3HXHDvrK7kCDxB0xspbDSxML4AX6AjT9gwScYS0xsLLwsmxgl4t3Mzpkz99w5M1cyVcV2GOv5hInJqekZ/2xgbn5hMRhaWi7aRtOSeUE2VMMqS6LNVUXnBUdxVF42LS5qkspLUmN/sF9qcctWDP3QaZu8qol1XTlRZNEhKl+uJWqhCIsxN8KjIO6BCLzIGKFHHOEYBmQ0oYFDh0NYhQibvgriYDCJq6JDnEVIcfc5zhEgbZOyOGWIxDboX6dVxWN1Wg9q2q5aplNUGhYpw4iyF3bP+uyZPbBX9vlnrY5bY+ClTbM01HKzFrxYzX/8q9JodnD6rRrr2cEJdlyvCnk3XWZwC3mob511+/ndXLSzwW7ZG/m/YT32RDfQW+/yXZbnrsf4kcgLvRg1KP67HaOgmIjFt2PJbDKS3vNa5cca1rFJ/UghjQNkUKD6dVziCl3BL8SELSE1TBV8nmYFP0JIfwESkpB9
X2
AAAClnichVHLSsNAFD2N73fUjeAmWCquZCKC4kJEEV22am2hLSGJow3Ny2Ra0eIP+AMuxIWCivgBfoAbf8CFnyAuFdy48DYNiIp6w2TOnLnnzpm5hm9boWDsMSG1tLa1d3R2dff09vUPyINDm6FXDUyeNT3bC/KGHnLbcnlWWMLmeT/gumPYPGdUlhr7uRoPQstzN8S+z0uOvuNa25apC6I0eTCvqUrR5rtKUZS50DVVk5NskkWh/ARqDJKII+3JtyhiCx5MVOGAw4UgbENHSF8BKhh84kqoExcQsqJ9jkN0k7ZKWZwydGIr9N+hVSFmXVo3aoaR2qRTbBoBKRWk2AO7Zi/snt2wJ/b+a616VKPhZZ9mo6nlvjZwNLL+9q/KoVmg/Kn607PANmYjrxZ59yOmcQuzqa8dHL+sz62l6uPsnD2T/zP2yO7oBm7t1bzI8LWTP/wY5IVejBqkfm/HT7A5NakSzkwnFxbjVnViFGOYoH7MYAGrSCNL9fdwiktcSSPSvLQsrTRTpUSsGcaXkNIfLpKWEA==
AAAClnichVHLSsNAFD3GV62vVjeCm2JRXMmkCooLKYrosg+rBSshiWMbTJOYTCu1+AP+gAtxoaAifoAf4MYfcOEniEsFNy68SQOiot4wmTNn7rlzZq7mmIYnGHtsk9o7Oru6Iz3R3r7+gcFYfGjds2uuzgu6bdpuUVM9bhoWLwhDmLzouFytaibf0HaX/P2NOnc9w7bWRMPhW1W1bBk7hq4KopRYvKikEiWT7yVKosKFqkwrsSSbYkEkfgI5BEmEkbFjtyhhGzZ01FAFhwVB2IQKj75NyGBwiNtCkziXkBHscxwiStoaZXHKUIndpX+ZVpsha9Har+kFap1OMWm4pExgnD2wa/bC7tkNe2Lvv9ZqBjV8Lw2atZaWO8rg0Uj+7V9VlWaByqfqT88CO5gLvBrk3QkY/xZ6S18/OH7Jz+fGmxPsnD2T/zP2yO7oBlb9Vb/I8tzJH3408kIvRg2Sv7fjJ1hPTcmEszPJ9GLYqghGMYZJ6scs0lhFBgWqv49TXOJKGpEWpGVppZUqtYWaYXwJKfMBNQCWEw==
AAAClnichVHLSsNAFD2Nr1pfrW4KbopFcSXTIigupCiiyz6sFqyEJE7bYF4m00ot/oA/4EJcKKiIH+AHuPEHXPgJ4lLBjQtv04CoqDdM5syZe+6cmas6hu4Jxh5DUld3T29fuD8yMDg0PBKNjW54dt3VeFGzDdstqYrHDd3iRaELg5cclyumavBNdXe5vb/Z4K6n29a6aDp821Sqll7RNUUQJUdjJTmdKBt8L1EWNS4UOS1Hk2yG+ZH4CVIBSCKIrB29RRk7sKGhDhMcFgRhAwo8+raQAoND3DZaxLmEdH+f4xAR0tYpi1OGQuwu/au02gpYi9btmp6v1ugUg4ZLygQm2QO7Zi/snt2wJ/b+a62WX6PtpUmz2tFyRx45ihfe/lWZNAvUPlV/ehaoYN73qpN3x2fat9A6+sbB8UthIT/ZmmLn7Jn8n7FHdkc3sBqv2kWO50/+8KOSF3oxalDqezt+go30TIpwbjaZWQpaFcY4JjBN/ZhDBmvIokj193GKS1xJcWlRWpFWO6lSKNCM4UtI2Q8y4JYS
✓1
✓2
AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTg2JcEd245CGPBAlp64ANpW3aQoLEHzBxKwtXmrgwfoAf4MYfcMEnGJeYuHHhpTQxSsTbTOfMmXvunJkrm5pqO4z1fMLE5NT0jH82MDe/sBgMLS3nbaNpKTynGJphFWXJ5pqq85yjOhovmhaXGrLGC3L9YLBfaHHLVg39yGmbvNyQarpaVRXJISpbrIiVUITFmBvhUSB6IAIvUkboEcc4gQEFTTTAocMhrEGCTV8JIhhM4sroEGcRUt19jnMESNukLE4ZErF1+tdoVfJYndaDmrarVugUjYZFyjCi7IXdsz57Zg/slX3+Wavj1hh4adMsD7XcrAQvVrMf/6oaNDs4/VaN9eygil3Xq0reTZcZ3EIZ6ltn3X52LxPtbLBb9kb+b1iPPdEN9Na7cpfmmesxfmTyQi9GDRJ/t2MU5Ldi4k4sno5Hkvteq/xYwzo2qR8JJHGIFHJUv4ZLXKEr+IWYsC0khqmCz9Os4EcIyS8QcpB8
X1
AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhAUIwV0caShzwSJGR3HXHDvrK7kCDxB0xspbDSxML4AX6AjT9gwScYS0xsLLwsmxgl4t3Mzpkz99w5M1cyVcV2GOv5hInJqekZ/2xgbn5hMRhaWi7aRtOSeUE2VMMqS6LNVUXnBUdxVF42LS5qkspLUmN/sF9qcctWDP3QaZu8qol1XTlRZNEhKl+uJWqhCIsxN8KjIO6BCLzIGKFHHOEYBmQ0oYFDh0NYhQibvgriYDCJq6JDnEVIcfc5zhEgbZOyOGWIxDboX6dVxWN1Wg9q2q5aplNUGhYpw4iyF3bP+uyZPbBX9vlnrY5bY+ClTbM01HKzFrxYzX/8q9JodnD6rRrr2cEJdlyvCnk3XWZwC3mob511+/ndXLSzwW7ZG/m/YT32RDfQW+/yXZbnrsf4kcgLvRg1KP67HaOgmIjFt2PJbDKS3vNa5cca1rFJ/UghjQNkUKD6dVziCl3BL8SELSE1TBV8nmYFP0JIfwESkpB9
X2
AAAClnichVHLSsNAFD2N73fUjeAmWCquZCKC4kJEEV22am2hLSGJow3Ny2Ra0eIP+AMuxIWCivgBfoAbf8CFnyAuFdy48DYNiIp6w2TOnLnnzpm5hm9boWDsMSG1tLa1d3R2dff09vUPyINDm6FXDUyeNT3bC/KGHnLbcnlWWMLmeT/gumPYPGdUlhr7uRoPQstzN8S+z0uOvuNa25apC6I0eTCvqUrR5rtKUZS50DVVk5NskkWh/ARqDJKII+3JtyhiCx5MVOGAw4UgbENHSF8BKhh84kqoExcQsqJ9jkN0k7ZKWZwydGIr9N+hVSFmXVo3aoaR2qRTbBoBKRWk2AO7Zi/snt2wJ/b+a616VKPhZZ9mo6nlvjZwNLL+9q/KoVmg/Kn607PANmYjrxZ59yOmcQuzqa8dHL+sz62l6uPsnD2T/zP2yO7oBm7t1bzI8LWTP/wY5IVejBqkfm/HT7A5NakSzkwnFxbjVnViFGOYoH7MYAGrSCNL9fdwiktcSSPSvLQsrTRTpUSsGcaXkNIfLpKWEA==
0CMJWJPVT5SFFTًحءُ涸圓鸡חז׷
鸐䌢ךⱄ䌓涸✳ⴓⶴ
Random forests ntree=10
Extra trees ntree=10

View Slide

寸㹀加ך♧菙⻉
x1
AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GXRjcuK9gHtUDJppg1NMkOSEcvYT3Cre3fi1p9x65eYtrPQ1gMXDufcy7mcMBHcWN//8gorq2vrG8XN0tb2zu5eef+gaeJUU9agsYh1OySGCa5Yw3IrWDvRjMhQsFY4up76rQemDY/VvR0nLJBkoHjEKbFOunvs4V654lf9GdAywTmpQI56r/zd7cc0lUxZKogxHewnNsiItpwKNil1U8MSQkdkwDqOKiKZCbLZqxN04pQ+imLtRlk0U39fZEQaM5ah25TEDs2iNxX/9UK5kGyjyyDjKkktU3QeHKUC2RhNe0B9rhm1YuwIoZq73xEdEk2odW2VXCl4sYJl0jyrYr+Kb88rtau8niIcwTGcAoYLqMEN1KEBFAbwDC/w6j15b9679zFfLXj5zSH8gff5A0z4lEY=
x2
AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFl047KifUA7lEyaaUOTzJBkxDL2E9zq3p249Wfc+iWm7Sy09cCFwzn3ci4nTAQ31vO+0Mrq2vrGZmGruL2zu7dfOjhsmjjVlDVoLGLdDolhgivWsNwK1k40IzIUrBWOrqd+64Fpw2N1b8cJCyQZKB5xSqyT7h571V6p7FW8GfAy8XNShhz1Xum7249pKpmyVBBjOr6X2CAj2nIq2KTYTQ1LCB2RAes4qohkJshmr07wqVP6OIq1G2XxTP19kRFpzFiGblMSOzSL3lT81wvlQrKNLoOMqyS1TNF5cJQKbGM87QH3uWbUirEjhGrufsd0SDSh1rVVdKX4ixUsk2a14nsV//a8XLvK6ynAMZzAGfhwATW4gTo0gMIAnuEFXtETekPv6GO+uoLymyP4A/T5A06MlEc=

View Slide

x1
x2
n0
AAAB+nicbVA9SwNBFHwXv2L8ilraLAbBKtyJoGXQxjKiiYHkCHubd8mS3b1jd08IMT/BVns7sfXP2PpL3CRXaOLAg2HmPeYxUSq4sb7/5RVWVtfWN4qbpa3tnd298v5B0ySZZthgiUh0K6IGBVfYsNwKbKUaqYwEPkTD66n/8Ija8ETd21GKoaR9xWPOqHXSner63XLFr/ozkGUS5KQCOerd8nenl7BMorJMUGPagZ/acEy15UzgpNTJDKaUDWkf244qKtGE49mrE3LilB6JE+1GWTJTf1+MqTRmJCO3KakdmEVvKv7rRXIh2caX4ZirNLOo2Dw4zgSxCZn2QHpcI7Ni5AhlmrvfCRtQTZl1bZVcKcFiBcukeVYN/Gpwe16pXeX1FOEIjuEUAriAGtxAHRrAoA/P8AKv3pP35r17H/PVgpffHMIfeJ8/O4iUOw==

dn0
AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8XPpr65QqyGw1UrdYgsmvIdd26IejCrTcc6NhojgpYouWXPwdhQrKYCk04VqrvoFR7OZaaEU6npUGmaIrJGA9p31CBY6q8fH7uFJ4ZJYRRIk0JDefqz4kcx0pN4sB0xliP1Ko3E//0gnhls47qXs5EmmkqyGJxlHGoEzjLAoZMUqL5xBBMJDO3QzLCEhNtEiuZUL4/h/+Tjms7yHZuq5Xm1TKeIjgBp+AcOOASNMENaIE2IGAMHsETeLYerBfr1XpbtBas5cwx+AXr/QsAXZZn

View Slide

x1
x2
n0

dn0
n1
AAAB+nicbVA9SwNBFHwXv2L8ilraLAbBKtyJoGXQxjKiiYHkCHubvWTJ7t6x+04IMT/BVns7sfXP2PpL3CRXaOLAg2HmPeYxUSqFRd//8gorq2vrG8XN0tb2zu5eef+gaZPMMN5giUxMK6KWS6F5AwVK3koNpyqS/CEaXk/9h0durEj0PY5SHira1yIWjKKT7nQ36JYrftWfgSyTICcVyFHvlr87vYRlimtkklrbDvwUwzE1KJjkk1InszylbEj7vO2oporbcDx7dUJOnNIjcWLcaCQz9ffFmCprRypym4riwC56U/FfL1ILyRhfhmOh0wy5ZvPgOJMEEzLtgfSE4QzlyBHKjHC/EzaghjJ0bZVcKcFiBcukeVYN/Gpwe16pXeX1FOEIjuEUAriAGtxAHRrAoA/P8AKv3pP35r17H/PVgpffHMIfeJ8/PRyUPA==

dn1
AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8XvjP1yxVkNxqoWq1BZNeQ67p1Q9CFW2840LHRHBWwRMsvfw7ChGQxFZpwrFTfQan2ciw1I5xOS4NM0RSTMR7SvqECx1R5+fzcKTwzSgijRJoSGs7VnxM5jpWaxIHpjLEeqVVvJv7pBfHKZh3VvZyJNNNUkMXiKONQJ3CWBQyZpETziSGYSGZuh2SEJSbaJFYyoXx/Dv8nHdd2kO3cVivNq2U8RXACTsE5cMAlaIIb0AJtQMAYPIIn8Gw9WC/Wq/W2aC1Yy5lj8AvW+xcB8pZo

View Slide

x1
x2
n0

dn0
`5
AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKosegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL2LXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d16pXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AI70lYk=
`6
AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKqMegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL2LXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d16pXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AJCIlYo=
n4
AAAB+nicbVDLSgMxFL2pr1pfVZdugkVwVWakoMuiG5cV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sZ73hQpr6xubW8Xt0s7u3v5B+fCoZeJUU9aksYh1JySGCa5Y03IrWCfRjMhQsHY4vpn57UemDY/Vg50kLJBkqHjEKbFOulf9Wr9c8areHHiV+DmpQI5Gv/zdG8Q0lUxZKogxXd9LbJARbTkVbFrqpYYlhI7JkHUdVUQyE2TzV6f4zCkDHMXajbJ4rv6+yIg0ZiJDtymJHZllbyb+64VyKdlGV0HGVZJapugiOEoFtjGe9YAHXDNqxcQRQjV3v2M6IppQ69oquVL85QpWSeui6ntV/65WqV/n9RThBE7hHHy4hDrcQgOaQGEIz/ACr+gJvaF39LFYLaD85hj+AH3+AEHYlD8=

dn4
AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSYxxXZXdOOygn1AG8JkMm2HTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+YcqY0Qh9WYW19Y3OruF3a2d3bPygfHrVVkklCWyThieyGWFHOBG1ppjntppLiOOS0E46vZ37nnkrFEnGnJyn1YzwUbMAI1kbqREEuAm8alCvIrteR51UhsqvIdd2aIejCrdUd6NhojgpYohmUP/tRQrKYCk04VqrnoFT7OZaaEU6npX6maIrJGA9pz1CBY6r8fH7uFJ4ZJYKDRJoSGs7VnxM5jpWaxKHpjLEeqVVvJv7phfHKZj2o+TkTaaapIIvFg4xDncBZFjBikhLNJ4ZgIpm5HZIRlphok1jJhPL9OfyftF3bQbZz61UaV8t4iuAEnIJz4IBL0AA3oAlagIAxeARP4Nl6sF6sV+tt0VqwljPH4Bes9y8GsZZr
n1

dn1

View Slide

x1
x2
n0

dn0
`5
`6
n4

dn4
n1

dn1
`1
AAAB/XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNlJJxkzM7vMzAphCf6CV717E69+i1e/xEmyB00saCiquqmmokRwY33/y1tZXVvf2CxsFbd3dvf2SweHDROnmmGdxSLWrYgaFFxh3XIrsJVopDIS2IxGN1O/+Yja8Fjd23GCoaQDxfucUeukRgeF6AbdUtmv+DOQZRLkpAw5at3Sd6cXs1SiskxQY9qBn9gwo9pyJnBS7KQGE8pGdIBtRxWVaMJs9u2EnDqlR/qxdqMsmam/LzIqjRnLyG1Kaodm0ZuK/3qRXEi2/asw4ypJLSo2D+6ngtiYTKsgPa6RWTF2hDLN3e+EDammzLrCiq6UYLGCZdI4rwR+Jbi7KFev83oKcAwncAYBXEIVbqEGdWDwAM/wAq/ek/fmvXsf89UVL785gj/wPn8AiKSVhQ==
`2
AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY9BLx4jmAckS5id9CZjZmeXmVkhLMFf8Kp3b+LVb/HqlzhJ9qCJBQ1FVTfVVJAIro3rfjmFtfWNza3idmlnd2//oHx41NJxqhg2WSxi1QmoRsElNg03AjuJQhoFAtvB+Gbmtx9RaR7LezNJ0I/oUPKQM2qs1OqhEP1av1xxq+4cZJV4OalAjka//N0bxCyNUBomqNZdz02Mn1FlOBM4LfVSjQllYzrErqWSRqj9bP7tlJxZZUDCWNmRhszV3xcZjbSeRIHdjKgZ6WVvJv7rBdFSsgmv/IzLJDUo2SI4TAUxMZlVQQZcITNiYgllitvfCRtRRZmxhZVsKd5yBaukVat6btW7u6jUr/N6inACp3AOHlxCHW6hAU1g8ADP8AKvzpPz5rw7H4vVgpPfHMMfOJ8/ijiVhg==
n2
AAAB+nicbVDLSgMxFL2pr1pfVZdugkVwVWaKoMuiG5cV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sZ73hQpr6xubW8Xt0s7u3v5B+fCoZeJUU9aksYh1JySGCa5Y03IrWCfRjMhQsHY4vpn57UemDY/Vg50kLJBkqHjEKbFOulf9Wr9c8areHHiV+DmpQI5Gv/zdG8Q0lUxZKogxXd9LbJARbTkVbFrqpYYlhI7JkHUdVUQyE2TzV6f4zCkDHMXajbJ4rv6+yIg0ZiJDtymJHZllbyb+64VyKdlGV0HGVZJapugiOEoFtjGe9YAHXDNqxcQRQjV3v2M6IppQ69oquVL85QpWSatW9b2qf3dRqV/n9RThBE7hHHy4hDrcQgOaQGEIz/ACr+gJvaF39LFYLaD85hj+AH3+AD6wlD0=

dn2
AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8Xvjv1yxVkNxqoWq1BZNeQ67p1Q9CFW2840LHRHBWwRMsvfw7ChGQxFZpwrFTfQan2ciw1I5xOS4NM0RSTMR7SvqECx1R5+fzcKTwzSgijRJoSGs7VnxM5jpWaxIHpjLEeqVVvJv7pBfHKZh3VvZyJNNNUkMXiKONQJ3CWBQyZpETziSGYSGZuh2SEJSbaJFYyoXx/Dv8nHdd2kO3cVivNq2U8RXACTsE5cMAlaIIb0AJtQMAYPIIn8Gw9WC/Wq/W2aC1Yy5lj8AvW+xcDh5Zp

View Slide

x1
x2
n0

dn0
`5
`6
n4

dn4
n1

dn1
`1
`2
n2

dn2
`3
AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyqoMegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL3zXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d1GpXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AIvMlYc=
`4
AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmAckS5id9CZjZmeXmVkhLMFf8Kp3b+LVb/HqlzhJ9qCJBQ1FVTfVVJAIro3rfjmFtfWNza3idmlnd2//oHx41NJxqhg2WSxi1QmoRsElNg03AjuJQhoFAtvB+Gbmtx9RaR7LezNJ0I/oUPKQM2qs1OqhEP1av1xxq+4cZJV4OalAjka//N0bxCyNUBomqNZdz02Mn1FlOBM4LfVSjQllYzrErqWSRqj9bP7tlJxZZUDCWNmRhszV3xcZjbSeRIHdjKgZ6WVvJv7rBdFSsgmv/IzLJDUo2SI4TAUxMZlVQQZcITNiYgllitvfCRtRRZmxhZVsKd5yBaukdVH13Kp3V6vUr/N6inACp3AOHlxCHW6hAU1g8ADP8AKvzpPz5rw7H4vVgpPfHMMfOJ8/jWCViA==
n3
AAAB+nicbVC7SgNBFL0bXzG+opY2g0GwCrsqaBm0sYxoHpAsYXYymwyZxzIzK4Q1n2CrvZ3Y+jO2fomTZAtNPHDhcM69nMuJEs6M9f0vr7Cyura+UdwsbW3v7O6V9w+aRqWa0AZRXOl2hA3lTNKGZZbTdqIpFhGnrWh0M/Vbj1QbpuSDHSc0FHggWcwItk66l73zXrniV/0Z0DIJclKBHPVe+bvbVyQVVFrCsTGdwE9smGFtGeF0UuqmhiaYjPCAdhyVWFATZrNXJ+jEKX0UK+1GWjRTf19kWBgzFpHbFNgOzaI3Ff/1IrGQbOOrMGMySS2VZB4cpxxZhaY9oD7TlFg+dgQTzdzviAyxxsS6tkqulGCxgmXSPKsGfjW4u6jUrvN6inAEx3AKAVxCDW6hDg0gMIBneIFX78l78969j/lqwctvDuEPvM8fQESUPg==

dn3
AAAB/nicdVDLSsNAFJ34rPVVdelmsAiuwiRtsd0V3bisYB/QhjCZTNqhk0mYmQglFPwFt7p3J279Fbd+idOHoEUPXDiccy/33hOknCmN0Ie1tr6xubVd2Cnu7u0fHJaOjjsqySShbZLwRPYCrChngrY105z2UklxHHDaDcbXM797T6ViibjTk5R6MR4KFjGCtZG6oZ8LvzL1S2VkNxqoWq1BZNeQ67p1Q1DFrTcc6NhojjJYouWXPgdhQrKYCk04VqrvoFR7OZaaEU6nxUGmaIrJGA9p31CBY6q8fH7uFJ4bJYRRIk0JDefqz4kcx0pN4sB0xliP1Ko3E//0gnhls47qXs5EmmkqyGJxlHGoEzjLAoZMUqL5xBBMJDO3QzLCEhNtEiuaUL4/h/+Tjms7yHZuq+Xm1TKeAjgFZ+ACOOASNMENaIE2IGAMHsETeLYerBfr1XpbtK5Zy5kT8AvW+xcFHJZq

View Slide

x1
x2
n0

dn0
`5
`6
n4

dn4
n1

dn1
`1
`2
n2

dn2
`3
`4
n3

dn3
ぐ걄㚖ד荈歋חխխխ׾㷕统
AAACq3ichVE9S8NQFD2N35+tugguxVKpg/IiguJUdHGs1rZilZLE1xqbL5K0WKt/wMlN1EnBQfwZLv0DDv0J4qjg4uBNGhAtrTck77xz77nvvFzZ0lTHZawZEnp6+/oHBoeGR0bHxsORicmsY1ZshWcUUzPtXVlyuKYaPOOqrsZ3LZtLuqzxnFze8PK5Krcd1TR23JrFD3SpZKhFVZFcorKpRO3sZL4QibFF5ke0HYgBiCGIlBlpYB+HMKGgAh0cBlzCGiQ49OQhgsEi7gB14mxCqp/nOMcwaStUxalCIrZM3xLt8gFr0N7r6fhqhU7R6LVJGUWcvbBH9s4a7Im9sq+Ovep+D89LjVa5peVWIXwxnf78V6XT6uLoR9XVs4siVn2vKnm3fMa7hdLSV0+v3tNr2/H6HLtnb+T/jjXZM93AqH4oD1t8+7aLH5m8dP5jXj6ooBGKfwfWDrJLiyLhreVYcj0Y5iBmMIsETWwFSWwihQydcIxLXONGWBDSwp6w3yoVQoFmCr9C4N89C5ka

View Slide

■ OC1/Oblique Tree (Murthy+ 1994)
■ Perceptron Tree (Utgoff , 1988)
■ Large Margin Tree (Wu+ 1999; Bennett+ 2000)
■ Margin Tree (Tibshirani & Hastie 2007)
■ Geometric Decision Tree (Manwani & Sastry,
2012)
■ HHCART (Wickramarachchi+, 2016)
■ M5/M5’ (Quinlan, 1992; Wang & Witten 1997)
■ RETIS (Karalic & Cestnik 1991)
■ SECRET (Dobra & Gehrke, 2002)
■ SMOTI (Malerba+ 2004)
■ MAUVE (Vens & Blockeel, 2006)
■ LLRT (Vogel, Asparouhov, Scheffer, 2007)
■ Cubist (Quinlan, 2011)
AAAClnichVHLSsNAFD3Gd3201Y3gplgqrmQiguJCiiK6rNZqoZWQxLEdmiYxmVa0+AP+gAtxoaAifoAf4MYfcOEniEsFNy68TQOiot4wmTNn7rlzZq7hWsKXjD22Ke0dnV3dPb2Rvv6BwWgsPrThOzXP5DnTsRwvb+g+t4TNc1JIi+ddj+tVw+KbRmWxub9Z554vHHtd7rt8q6qXbLEjTF0SpcXieU0kiiW+myjKMpe6pmqxJJtkQSR+AjUESYSRcWK3KGIbDkzUUAWHDUnYgg6fvgJUMLjEbaFBnEdIBPsch4iQtkZZnDJ0Yiv0L9GqELI2rZs1/UBt0ikWDY+UCaTYA7tmL+ye3bAn9v5rrUZQo+lln2ajpeWuFj0ayb79q6rSLFH+VP3pWWIHs4FXQd7dgGnewmzp6wfHL9m5tVRjnJ2zZ/J/xh7ZHd3Arr+aF6t87eQPPwZ5oRejBqnf2/ETbExNqoRXp5PphbBVPRjFGCaoHzNIYwUZ5Kj+Hk5xiStlRJlXlpTlVqrSFmqG8SWUzAedy5ZD
AAAClnichVHLSsNAFD2Nr/psq5uCm2BRXMlEBMWFFEV02VarBSshiWONTZOYTCu1+AP+gAtxoaAifoAf4MYfcOEniEsFNy68TQOiot4wmTNn7rlzZq7uWqYvGHuMSG3tHZ1d0e6e3r7+gVg8MbjmO1XP4HnDsRyvoGs+t0yb54UpLF5wPa5VdIuv6+WF5v56jXu+6dirou7yzYpWss1t09AEUWo8UVB35WKJ78lFscOFpk6q8RSbYEHIP4ESghTCyDjxWxSxBQcGqqiAw4YgbEGDT98GFDC4xG2iQZxHyAz2OQ7RQ9oqZXHK0Igt079Eq42QtWndrOkHaoNOsWh4pJQxyh7YNXth9+yGPbH3X2s1ghpNL3Wa9ZaWu2rsKLny9q+qQrPAzqfqT88C25gJvJrk3Q2Y5i2Mlr52cPyyMpsbbYyxc/ZM/s/YI7ujG9i1V+Miy3Mnf/jRyQu9GDVI+d6On2BtckIhnJ1KpefDVkUxjBGMUz+mkcYyMshT/X2c4hJXUlKakxalpVaqFAk1Q/gSUuYDohmWRQ==
AAAClnichVHLSsNAFD3Gd3201Y3gplgqrmSqguJCRBFd1mof0JaQxLENTZOYTCta/AF/wIW4UFARP8APcOMPuPATxGUFNy68TQOixXrDZM6cuefOmbmqbeiuYOylS+ru6e3rHxgMDA2PjAZD4bG0a1Udjac0y7CcrKq43NBNnhK6MHjWdrhSUQ2eUcvrzf1MjTuubpm74sjmhYpSNPV9XVMEUXIonJXLkXyRH0TyosSFIs/LoSibZV5E2kHcB1H4kbBCD8hjDxY0VFEBhwlB2IACl74c4mCwiSugTpxDSPf2OU4QIG2VsjhlKMSW6V+kVc5nTVo3a7qeWqNTDBoOKSOIsWd2xxrsid2zV/b5Z626V6Pp5YhmtaXlthw8ndj5+FdVoVmg9K3q6FlgH0ueV5282x7TvIXW0teOzxo7y8lYfZpdsTfyf8le2CPdwKy9a9fbPHnewY9KXujFqEHx3+1oB+m52Tjh7YXo6prfqgFMYgoz1I9FrGILCaSo/iEucINbaUJakTakzVaq1OVrxvEjpMQXpmeWRw==
AAAClnichVHLSsNAFD2Nr1pfrW4EN8VScSVTKSgupCiiy1atLbQlJHFsg2kSk2lFiz/gD7gQFwoq4gf4AW78ARf9BHGp4MaFt2lAtFhvmMyZM/fcOTNXtQ3dFYw1A1JPb1//QHAwNDQ8MjoWjozvuFbN0XhWswzLyauKyw3d5FmhC4PnbYcrVdXgOXV/tbWfq3PH1S1zWxzZvFRVyqa+p2uKIEoOR/KyES2W+UG0KCpcKHJSDsfYHPMi2gkSPojBj7QVfkARu7CgoYYqOEwIwgYUuPQVkACDTVwJDeIcQrq3z3GCEGlrlMUpQyF2n/5lWhV81qR1q6brqTU6xaDhkDKKOHtmd+yNPbF79sI+/6zV8Gq0vBzRrLa13JbHTie3Pv5VVWkWqHyrunoW2MOi51Un77bHtG6htfX147O3raXNeGOGXbFX8n/JmuyRbmDW37XrDN887+JHJS/0YtSgxO92dIKd+bkE4UwyllrxWxXEFKYxS/1YQAobSCNL9Q9xgRvcSpPSsrQmrbdTpYCvmcCPkNJfqrWWSQ==
AAACmnichVHLSsNAFD3Gd31VRRB0USxVQZCJCIorqRvFjVprC1pCEqc1mBfJtKDBH/AHXLiq4EL7AX6AG3/AhZ8gLiu4ceFtGhAt1hsmc+bMPXfOzNVc0/AFYy8dUmdXd09vX39sYHBoeCQ+OnbgO2VP51ndMR0vr6k+Nw2bZ4UhTJ53Pa5amslz2ulGYz9X4Z5vOPa+OHN5wVJLtlE0dFUQpcQncoo8lzjSrCB/kVggwIWqyEo8yRZZGIlWIEcgiSh2nPgDjnAMBzrKsMBhQxA2ocKn7xAyGFziCgiI8wgZ4T7HBWKkLVMWpwyV2FP6l2h1GLE2rRs1/VCt0ykmDY+UCaTYM7tjdfbEauyVff5ZKwhrNLyc0aw1tdxVRi4nMx//qiyaBU6+VW09CxSxGno1yLsbMo1b6E195fyqnlnbSwWz7Ia9kf8qe2GPdAO78q7f7vK96zZ+NPJCL0YNkn+3oxUcLC3KhHeXk+vpqFV9mMIM5qkfK1jHJnaQpfoBqrhHTZqW0tKWtN1MlToizTh+hLT/BZu5lww=
AAACmnichVHLSsNAFD3G97NVEQRdFIsPEMpUBcVVqRvFTVutLVQJSZzW0LxIpgUN/oA/4MKVggv1A/wAN/6Ai36CuFRw48LbNCBa1Bsmc+bMPXfOzFUdQ/cEY40OqbOru6e3r39gcGh4JBIdHdvz7Jqr8bxmG7ZbVBWPG7rF80IXBi86LldM1eAFtbrR3C/UuevptrUrjh1+YCoVSy/rmiKIkqMTBXl5Pravmn7xNLZIgAtFXpajcZZgQcTaQTIEcYSRsaP32MchbGiowQSHBUHYgAKPvhKSYHCIO4BPnEtID/Y5TjFA2hplccpQiK3Sv0KrUshatG7W9AK1RqcYNFxSxjDLntgNe2WP7I49s49fa/lBjaaXY5rVlpY7cuRscuf9X5VJs8DRl+pPzwJlrAVedfLuBEzzFlpLXz85f91Zz836c+yKvZD/S9ZgD3QDq/6mXWd57uIPPyp5oRejBiV/tqMd7C0lkoSzK/FUOmxVH6YwgwXqxypS2EQGearv4xK3uJOmpbS0JW23UqWOUDOObyHtfgKkXZcQ
AAACmnichVHLSsNAFD2N7/qqiiDoolh8gFCmRVBciW4UN9W2tqASkjitwbxIpgUN/oA/4MJVBRfaD/AD3PgDLvoJ4lLBjQtv04Bosd4wmTNn7rlzZq7qGLonGGtEpK7unt6+/oHo4NDwyGhsbHzfsyuuxvOabdhuUVU8bugWzwtdGLzouFwxVYMX1NPN5n6hyl1Pt62cOHP4kamULb2ka4ogSo5NFuT0QvxQNf3iRXyJABeKnJZjCZZkQcTbQSoECYSRsWMPOMQxbGiowASHBUHYgAKPvgOkwOAQdwSfOJeQHuxzXCBK2gplccpQiD2lf5lWByFr0bpZ0wvUGp1i0HBJGccce2Z37I09sTp7YZ9/1vKDGk0vZzSrLS135NHLqezHvyqTZoGTb1VHzwIlrAZedfLuBEzzFlpLXz2/esuu7c358+yGvZL/GmuwR7qBVX3Xbnf53nUHPyp5oRejBqV+t6Md7KeTKcK7y4n1jbBV/ZjGLBapHytYxxYyyFN9HzXcoy7NSBvStrTTSpUioWYCP0LKfQGgC5cO
AAACmnichVHLSsNAFD2N7/qqiiDoolh8gFCmUlBciW4UN9W2tqASkjitwbxIpgUN/oA/4MJVBRfaD/AD3PgDLvoJ4lLBjQtv04Bosd4wmTNn7rlzZq7qGLonGGtEpK7unt6+/oHo4NDwyGhsbHzfsyuuxvOabdhuUVU8bugWzwtdGLzouFwxVYMX1NPN5n6hyl1Pt62cOHP4kamULb2ka4ogSo5NFuT0QvxQNf3iRXyJABeKnJZjCZZkQcTbQSoECYSRsWMPOMQxbGiowASHBUHYgAKPvgOkwOAQdwSfOJeQHuxzXCBK2gplccpQiD2lf5lWByFr0bpZ0wvUGp1i0HBJGccce2Z37I09sTp7YZ9/1vKDGk0vZzSrLS135NHLqezHvyqTZoGTb1VHzwIlrAZedfLuBEzzFlpLXz2/esuu7c358+yGvZL/GmuwR7qBVX3Xbnf53nUHPyp5oRejBqV+t6Md7C8nU4R304n1jbBV/ZjGLBapHytYxxYyyFN9HzXcoy7NSBvStrTTSpUioWYCP0LKfQGor5cS
AAACmnichVHLSsNAFD3G97NVEQRdFIsPEMpUFMVVqRvFTVutLVQJSZzW0LxIpgUN/oA/4MKVggv1A/wAN/6Ai36CuFRw48LbNCBa1Bsmc+bMPXfOzFUdQ/cEY40OqbOru6e3r39gcGh4JBIdHdvz7Jqr8bxmG7ZbVBWPG7rF80IXBi86LldM1eAFtbrR3C/UuevptrUrjh1+YCoVSy/rmiKIkqMTBXllPravmn7xNLZIgAtFXpGjcZZgQcTaQTIEcYSRsaP32MchbGiowQSHBUHYgAKPvhKSYHCIO4BPnEtID/Y5TjFA2hplccpQiK3Sv0KrUshatG7W9AK1RqcYNFxSxjDLntgNe2WP7I49s49fa/lBjaaXY5rVlpY7cuRscuf9X5VJs8DRl+pPzwJlrAVedfLuBEzzFlpLXz85f91Zz836c+yKvZD/S9ZgD3QDq/6mXWd57uIPPyp5oRejBiV/tqMd7C0lkoSzy/FUOmxVH6YwgwXqxypS2EQGearv4xK3uJOmpbS0JW23UqWOUDOObyHtfgKtAZcU
AAACnnichVG7SgNBFD2u7/hI1EaxWQw+qjArgmIlCmIjGjUmYGTZXSfJkn25OwloEHt/wMJKwUIELf0AG3/Awk8QSwUbC282C6JivMvsnDlzz50zc3XPMgPB2FOL1NrW3tHZ1R3r6e3rjycGBrcDt+IbPGO4luvndC3glunwjDCFxXOezzVbt3hWLy/V97NV7gem62yJA4/v2lrRMQumoQmi1MRIVlUm5bxu13JHcr7I9+W8KHGhqYqaSLIUC0P+DZQIJBHFupu4Qx57cGGgAhscDgRhCxoC+naggMEjbhc14nxCZrjPcYQYaSuUxSlDI7ZM/yKtdiLWoXW9ZhCqDTrFouGTUsY4e2RX7JU9sGv2zD7+rFULa9S9HNCsN7TcU+Mnw5vv/6psmgVKX6qmngUKmAu9muTdC5n6LYyGvnp4+ro5vzFem2AX7IX8n7Mndk83cKpvxmWab5w18aOTF3oxapDysx2/wfZ0SiGcnkkuLEat6sIoxjBF/ZjFAlawjgzVP8YlbnArydKytCqtNVKllkgzhG8h5T4BYfKZHA==
AAACnnichVG7SgNBFD2u72eiNorNYvBRyWwQFCtREBvxmQcksuyuk2RxX+5OAhrE3h+wsFKwEEFLP8DGH7DIJ4hlBBsLbzYLoqLeZXbOnLnnzpm5umeZgWCs1iK1trV3dHZ19/T29Q/E4oND6cAt+wZPGa7l+lldC7hlOjwlTGHxrOdzzdYtntH3lxv7mQr3A9N1dsShx3dtreiYBdPQBFFqfDSjJqfkvG5Xs8dyvsgP5LwocaGpSTWeYDMsDPknUCKQQBQbbvweeezBhYEybHA4EIQtaAjoy0EBg0fcLqrE+YTMcJ/jGD2kLVMWpwyN2H36F2mVi1iH1o2aQag26BSLhk9KGRPsiV2zOntkN+yZvf9aqxrWaHg5pFlvarmnxk5Htt/+Vdk0C5Q+VX96FihgPvRqkncvZBq3MJr6ytFZfXtha6I6yS7ZC/m/YDX2QDdwKq/G1SbfOv/Dj05e6MWoQcr3dvwE6eSMQnhzNrG4FLWqC2MYxzT1Yw6LWMUGUlT/BFe4xZ0kSyvSmrTeTJVaIs0wvoSU/QBmSJke
AAACnnichVG7SgNBFD2u7/iK2ig2i8FHFSZGUKxEQWzEZ0zAhGV3ncQl+3J3EtAg9v6AhZVCChG09ANs/AELP0EsFWwsvNksiAbjXWbnzJl77pyZq7mm4QvGnluk1rb2js6u7khPb1//QHRwaNd3Sp7OU7pjOl5GU31uGjZPCUOYPON6XLU0k6e14nJtP13mnm849o44cnnOUgu2kTd0VRClREfTSnJKzmpWJXMiZwv8UM6KAy5UJalEYyzOgpAbQSIEMYSx4UTvkcU+HOgowQKHDUHYhAqfvj0kwOASl0OFOI+QEexznCBC2hJlccpQiS3Sv0CrvZC1aV2r6QdqnU4xaXiklDHBntg1e2OP7Ia9sM8/a1WCGjUvRzRrdS13lYGzke2Pf1UWzQIH36qmngXymA+8GuTdDZjaLfS6vnx8/ra9sDVRmWRX7JX8X7Jn9kA3sMvvenWTb1008aORF3oxalDidzsawe5MPEF4cza2uBS2qgtjGMc09WMOi1jFBlJU/xRV3OJOkqUVaU1ar6dKLaFmGD9CynwBap6ZIA==
AAACnnichVG7SgNBFD1Z389EbRSbxeCjkokEFCtREBvRJOYBiSy76yRZ3Je7k4AGsfcHLKwULETQ0g+w8Qcs/ASxVLCx8GazICrqXWbnzJl77pyZq7mm4QvGHiNSW3tHZ1d3T29f/8BgNDY0nPOdmqfzrO6YjlfQVJ+bhs2zwhAmL7geVy3N5Hltd6W5n69zzzcce0vsu3zbUiu2UTZ0VRClxMbySnJaLmlWo3Aolyp8Ty6JKheqklRicTbLgpB/gkQI4ghj04ndooQdONBRgwUOG4KwCRU+fUUkwOASt40GcR4hI9jnOEQvaWuUxSlDJXaX/hVaFUPWpnWzph+odTrFpOGRUsYke2CX7IXdsyv2xN5/rdUIajS97NOstbTcVaLHo5m3f1UWzQLVT9WfngXKWAi8GuTdDZjmLfSWvn5w8pJZTE82ptg5eyb/Z+yR3dEN7PqrfpHi6dM//GjkhV6MGpT43o6fIDc3myCcSsaXlsNWdWMcE5ihfsxjCWvYRJbqH+EC17iRZGlVWpc2WqlSJNSM4EtIhQ9u9Jki
AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhVmRYlYBW0s8zAPiCHsrpO4ZF/sbgIx+AOCrSmsFCzED/ADbPwBi3yCWEawsfBmsyAajHeZnTNn7rlzZq5saarjMtYLCBOTU9MzwdnQ3PzCYjiytFxwzKat8LxiaqZdkiWHa6rB867qarxk2VzSZY0X5cbBYL/Y4rajmsaR27Z4RZfqhlpTFcklKqdUxWokxhLMi+goEH0Qgx9pM/KIY5zAhIImdHAYcAlrkODQV4YIBou4CjrE2YRUb5/jHCHSNimLU4ZEbIP+dVqVfdag9aCm46kVOkWjYZMyijh7Yfesz57ZA3tln3/W6ng1Bl7aNMtDLbeq4YvV3Me/Kp1mF6ffqrGeXdSw63lVybvlMYNbKEN966zbz+1l450NdsveyP8N67EnuoHRelfuMjx7PcaPTF7oxahB4u92jILCVkIknNmOpfb9VgWxhnVsUj+SSOEQaeSpfh2XuEJXCAoJYUdIDlOFgK9ZwY8QUl8k/JB+
AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmQYlYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYPCBOTU9MzwdnQ3PzCYjiytFyyjZYl86JsqIZVkUSbq4rOi47iqLxiWlzUJJWXpea+u19uc8tWDP3Q6Zi8pokNXTlRZNEhqiDXU/VIjCWYF9FRkPRBDH5kjcgjjnAMAzJa0MChwyGsQoRNXxVJMJjE1dAlziKkePsc5wiRtkVZnDJEYpv0b9Cq6rM6rd2atqeW6RSVhkXKKOLshd2zAXtmD+yVff5Zq+vVcL10aJaGWm7WwxerhY9/VRrNDk6/VWM9OzjBjudVIe+mx7i3kIf69llvUNjNx7sb7Ja9kf8b1mdPdAO9/S7f5Xj+eowfibzQi1GDkr/bMQpKqUSScG4rltnzWxXEGtaxSf1II4MDZFGk+g1c4go9ISgkhG0hPUwVAr5mBT9CyHwBJxyQfw==
AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmPohYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYLCGPjE5NTwenQzOzcfDiysFi0jaYl84JsqIZVlkSbq4rOC47iqLxsWlzUJJWXpMa+u19qcctWDP3QaZu8qol1XTlRZNEhKi/XNmuRGEswL6LDIOmDGPzIGJFHHOEYBmQ0oYFDh0NYhQibvgqSYDCJq6JDnEVI8fY5zhEibZOyOGWIxDboX6dVxWd1Wrs1bU8t0ykqDYuUUcTZC7tnffbMHtgr+/yzVser4Xpp0ywNtNyshS+W8x//qjSaHZx+q0Z6dnCCHc+rQt5Nj3FvIQ/0rbNuP7+bi3fW2C17I/83rMee6AZ6612+y/Lc9Qg/EnmhF6MGJX+3YxgUNxJJwtmtWHrPb1UQK1jFOvUjhTQOkEGB6tdxiSt0haCQELaF1CBVCPiaJfwIIf0FKTyQgA==
AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmEolYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYPCBOTU9MzwdnQ3PzCYjiytFyyjZYl86JsqIZVkUSbq4rOi47iqLxiWlzUJJWXpea+u19uc8tWDP3Q6Zi8pokNXTlRZNEhqiDXU/VIjCWYF9FRkPRBDH5kjcgjjnAMAzJa0MChwyGsQoRNXxVJMJjE1dAlziKkePsc5wiRtkVZnDJEYpv0b9Cq6rM6rd2atqeW6RSVhkXKKOLshd2zAXtmD+yVff5Zq+vVcL10aJaGWm7WwxerhY9/VRrNDk6/VWM9OzjBjudVIe+mx7i3kIf69llvUNjNx7sb7Ja9kf8b1mdPdAO9/S7f5Xj+eowfibzQi1GDkr/bMQpKW4kk4VwqltnzWxXEGtaxSf1II4MDZFGk+g1c4go9ISgkhG0hPUwVAr5mBT9CyHwBK1yQgQ==
AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmYohYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYPCBOTU9MzwdnQ3PzCYjiytFyyjZYl86JsqIZVkUSbq4rOi47iqLxiWlzUJJWXpea+u19uc8tWDP3Q6Zi8pokNXTlRZNEhqiDXU/VIjCWYF9FRkPRBDH5kjcgjjnAMAzJa0MChwyGsQoRNXxVJMJjE1dAlziKkePsc5wiRtkVZnDJEYpv0b9Cq6rM6rd2atqeW6RSVhkXKKOLshd2zAXtmD+yVff5Zq+vVcL10aJaGWm7WwxerhY9/VRrNDk6/VWM9OzjBjudVIe+mx7i3kIf69llvUNjNx7sb7Ja9kf8b1mdPdAO9/S7f5Xj+eowfibzQi1GDkr/bMQpKW4kk4dx2LLPntyqINaxjk/qRRgYHyKJI9Ru4xBV6QlBICCkhPUwVAr5mBT9CyHwBLXyQgg==
ٌرٕ加 .PEFMUSFF

衝ظ٦سָ㹀侧✮庠̔简䕎✮庠
㢳㢌ꆀ加 .VMUJWBSJBUFUSFF

ⴓⶴָ⽃㢌ꆀ̔㢳㢌ꆀ简䕎

View Slide

ٌرٕ加 .PEFM5SFFT
衝ظ٦سָ㹀侧✮庠̔简䕎✮庠
㢳㢌ꆀ加 .VMUJWBSJBUF5SFFT
ⴓⶴָ⽃㢌ꆀ̔㢳㢌ꆀ简䕎
Regression Trees M5 MARS
㢳㢌ꆀ黝䘔涸㔐䌓
أفٓ؎ٝ ."34

ر٦ة 㔐䌓

ر٦ة ⴓ겲
Classiﬁcation Trees OC1 Multivariate CART
寸㹀加ד걄㚖׾✳ⴓ
ⶴׅ׷➿׻׶ח걄㚖
׾䫓׶刼־׷կ
(SFFEZד㷕统

View Slide

֮ת׶㹋欽⻉׃גְזְ˘
Ԩ ٌرٕ加衝ח 㹀侧✮庠״׶
醱꧟זٌرٕ׾䭯׋ׇ׷
Ӎ 걄㚖㞮歲ָ衼׃ֻ♶㸜㹀٥ꬊ鸬竲חז׷
˖ ..ˏך䎂徽⻉ؼُ٦ٔأذ؍ؙأ
˖ 㢳㢌ꆀ黝䘔㘗㔐䌓أفٓ؎ٝ ."34
ח״׷䫓׸简㘗⼒ⴓ简䕎㷕统
˖ 然桦涸寸㹀加
Ӎ 剑葺ⴓⶴ挿䱱稊ח衝ٌرٕךؿ؍حزָ䗳銲הז׶
׍ׯ׿הず儗ח剑黝⻉ׅ׷ה鎘皾؝أزָꬊ䌢ח㣐ֹֻז׷
㼎Ⳣ
Ԩ 㢳㢌ꆀ加✳ⴓⶴח ⽃㢌ꆀ✳ⴓⶴ״׶
醱꧟ז✳⦼ⴓ겲㐻׾欽ְ׷
Ӎ ر٦ة倖晙⻉ぐ걄㚖ך؟ٝفٕ侧幾㼰
׾饯ֿ׃麓ⶱ黝さ׃װְׅ
̔㢳ⴓ䀄דכזֻג✳ⴓ䀄ك٦أָ⚺崧זך׮ֿך椚歋ח״׷
Ӎ ⴓⶴח״׶ぐ걄㚖ח衅׍׷؟ٝفٕ侧כו׿ו׿幾׷ךד如⯋׾
幾׵ׁזְꣲ׶ծ㼭؟ٝفٕ鏣㹀ד葺ְⴓⶴך㷕统ָ㔭ꨇחז׷
أفٓ؎ٝ
걄㚖㞮歲ד鸬竲
˟ֿ׸כ㷕统倯䒭ָ
խ(SFFEZזְׇ

View Slide

劤䔲ח⼒ⴓ涸㹀侧٥鯥⚛遤✳ⴓⶴד葺ְךַכ剑㣐ךꟼ䗰
y=0.89 + -0.36 max(0,x-0.99)
+ -0.88 max(0,0.99-x)
y=1.23 + -0.59 max(0,x-0.99)
+ -1.32 max(0,0.99-x)
+ 1.41 max(0,x-4.96)
y=0.98 + -0.23 max(0,x-0.99)
+ -1.0 max(0,0.99-x)
+ 1.69 max(0,x-4.96)
+ -0.51 max(0,x-2.31)
y=0.95 + -0.12 max(0,x-0.99)
+ -0.96 max(0,0.99-x)
+ 0.77 max(0,x-4.96)
+ -0.79 max(0,x-2.31)
+ 0.87 max(0,x-2.31)
y=0.66 + -0.29 max(0,x-0.99)
䠣湫חװ׷ה鎘皾儗꟦ַַָ׷ךד㹋ꥷך㷕统חכ
'BTU."34 'SJFENBO
ָ״ֻ欽ְ׵׸׷
Friedman JH. Multivariate Adaptive Regression Splines. The Annals of Statistics, 1991;19: 1–67.

View Slide

劤䔲ח⼒ⴓ涸㹀侧٥鯥⚛遤✳ⴓⶴד葺ְךַכ剑㣐ךꟼ䗰


鯥⚛遤זⱄ䌓涸✳ⴓⶴ
(SFFEZ䱱稊
⼒ⴓ涸չ㹀侧պ
׋׌ךؽٝ〳㢌זؼأزؚٓي

˖ ."34ך״ֲזأفٓ؎ٝ 䫓׸꬗
ך湫䱸ךؿ؍حذ؍ؚٝכ㹋ꥷח⢪ֲהꨇָ֮׷
걄㚖ⴓⶴה걄㚖XJTFזⱖ⫷ךず儗剑黝⻉כװכ׶♧菙חכהג׮ꨇ׃ְ

˖ 帾㾴㷕统ָ."40ךさ䧭ד剅ֽ׷QJFDFXJTFBOFז✮庠ה罋ִ׸לծػًٓة剑黝⻉׌ֽ
ׅ׸לչ걄㚖ⴓⶴךקֲכ⹧䩛ח㹀ת׷պ."4חז׏גְ׷挿כꬊ䌢ח莆㄂帾ְ

View Slide

ٓٝتي㼗䕦加3BOEPN1SPKFDUJPO5SFFT
STOC 2008 NIPS 2010
Google scholar citations: 419 Google scholar citations: 10
瑞꟦ⴓⶴر٦ة圓鸡ה׃גכ寸㹀加כLEUSFF׌ָ넝如⯋דכ31加ךקֲָ䚍颵ָ葺ְכ׆

View Slide

⼒ⴓ涸㹀侧׌ה 4QMJOFה殯ז׶
걄㚖㞮歲ָ♶鸬竲ֺׅ׷˘
RPܾఆ໦ͷྖҬ෼ׂ
10 RP vectors
֮׷ְכ♶鸬竲䚍ך㉏겗כչ،ٝ؟ٝـٕպד鍑嶊ׅ׷ַ׵孡ח׃זֻג葺ְךַتًזךַ
"OFأفٓ؎ٝח⡂׋䠬ׄך걄㚖ⴓⶴכ
㺁僒ח〳腉

View Slide

0QFORVFTUJPOT
˖ 植㖈ꬊ䌢ח״ֻ⢪׻׸׷寸㹀加،ٝ؟ٝـٕכQJFDFXJTFDPOTUBUד걄㚖圓眠כ
(SFFEZ鯥⚛遤✳ⴓⶴ
ֿ̔׸ָ植㹋涸ז剑׮葺ְ㧅⼿鍑זךַծ何㊣ׅ׷،؎ر،ָ֮׶䖤׷ךַ
˖ 315SFF&YUSB5SFFT(SBEJFOU#PPTUJOH
˖ ぐ걄㚖׀הח"OFזⱖ⫷׾㹀纏ׅ׷ךכ銲嗚鎢
˖ ⼒ⴓ涸㹀侧ךתתָ穗꿀ⴓ䋒ח屟׏גְג葺ְ
&YUSB5SFFTד亻⡂涸ז鸬竲䚍׾邌植ׅ׷

View Slide

ֿ׸כ//MJLFז寸㹀加圓眠װ寸㹀加ך//MJLFז㷕统חꟼ⤘
Popov S, Morozov S, Babenko A. Neural Oblivious Decision Ensembles for Deep Learning on
Tabular Data. ICLR 2020, http://arxiv.org/abs/1909.06312
Kontschieder P, Fiterau M, Criminisi A, Bulo SR. Deep Neural Decision Forests. IJCAI 2016,
https://www.ijcai.org/Proceedings/16/Papers/628.pdf
˖ %JFSFOUJBCMFז寸㹀加/FVSBM/FUXPSLTַ׵ך寸㹀加圓眠
Lee G-H, Jaakkola TS. Oblique Decision Trees from Derivatives of ReLU Networks.
ICLR 2020, https://openreview.net/pdf?id=Bke8UR4FPB
Lay N, Harrison AP, Schreiber S, Dawer G, Barbu A. Random Hinge Forest for Differentiable
Learning. ICML 2018, http://arxiv.org/abs/1802.03882
Zantedeschi V, Kusner MJ, Niculae V. Learning Binary Decision Trees by Argmin Differentiation.
ICML 2021, http://arxiv.org/abs/2010.04627
Hazimeh H, Ponomareva N, Mol P, Tan Z, Mazumder R. The Tree Ensemble Layer: Differentiability
meets Conditional Computation. ICML 2020, https://arxiv.org/abs/2002.07772

View Slide

הصُ٦ٕٓطحز

View Slide

⠓陽䖓ך荈䊹'PMMPXVQ
Thanks to ⸇秛륊♧ׁ׿ %/"/**

/(#PPTUBOE1SFEJDUJPO*OUFSWBMT
6ODFSUBJOUZ2VBOUJDBUJPO 62

"MFBUPSJDVODFSUBJOUZה&QJTUFNJDVODFSUBJOUZ
%JTUSJCVUJPO'SFF62JO.-
$POGPSNBM1SFEJDUJPO

View Slide

/(#PPTU
https://stanfordmlgroup.github.io/projects/ngboost/

View Slide

/(#PPTUBOE1SFEJDUJPO*OUFSWBMT
https://towardsdatascience.com/interpreting-the-probabilistic-predictions-from-ngboost-868d6f3770b2
NGBRegressor

View Slide

׍ז׫ח̕ך״ֲז׉׮׉׮锷׮㣐ְח֮׷˘
ֿ׸ז׵ה׮ַֻ˘ ׉׮׉׮锷ה׃גծ窟鎘㷕涸חכ؟ٝفٕ♶駈ד
֮׷6OEFSTQFDJFEז朐屣ד⡦ַ䒉鏣涸ז陽锷
כ〳腉זךַ
堣唒㷕统ٌرٕךEFQMPZ䖓ח⳿⠓ֲر٦ة ذأ
زر٦ة
כ搀ꣲծ鎮箺ر٦ة٥嗚鏾ر٦ةכ剣
ꣲծהְֲ搀椚鏣㹀דכ穠㽷չٌرٕך䌓秛غ
؎،أ JOEVDUJWFCJBT
պָ湡⵸ך㉏겗חوحث
ׅ׷ַ׌ָֽꅾ銲
ֿך䠐㄂ד׮ٌرٕך䮙⹛ך帾ְ椚鍑ה
׉׸׾㹋欽خ٦ٕה㹋㉏겗ח鼧⯋ׅ׷
㹋騧ךJUFSBUJPOכ㣐✲

View Slide

6ODFSUBJOUZ2VBOUJGJDBUJPO 62

堣唒㷕统ךչ✮庠պ׉ך׮ךכر٦ةך،َװ㉏겗鏣㹀׉ך׮ךךꣲ歲ח״׏ג姻然חכ
寸׃גז׵זְկ׬׃׹넝如⯋䚍זוך㔭ꨇׁ׾驎תִ׷ה䌢ח♧㹀ך然䏝ךꣲ歲ָ֮׷ה
罋ִ׷ץֹկ
堣唒㷕统✮庠ךչ♶然㹋䚍պ鐰⣣ ♶然ַׁך㹀ꆀ⻉
כ㹋欽堣唒㷕统ךꅾ銲ז㉏겗
Uncertainty Quantiﬁcation (UQ)
https://en.wikipedia.org/wiki/Uncertainty_quantiﬁcation
窟鎘㷕׌ֽדכזֻ鎘庠װ侧椚ٌرؚٔٝזו䊨㷕걄㚖׮סֻ׿ד
圫ղח陽锷ׁ׸ծׁתׂתז62䩛岀ָ䲿周ׁ׸גֹ׋կ

View Slide

"MFBUPSJDVODFSUBJOUZה&QJTUFNJDVODFSUBJOUZ
6ODFSUBJOUZ ♶然㹋䚍
חכչ"MFBUPSJDպז׮ךהչ&QJTUFNJDպז׮ךָ֮׶ծ圫ղזⴓꅿד
陽锷ׁ׸גֹ׋կ✳珏겲ך♶然㹋䚍ָ幉さׁ׸ג✮庠ٌرؚٔٝח䕦갟ׅ׷կ
https://en.wikipedia.org/wiki/Uncertainty_quantiﬁcation#Aleatoric_and_epistemic
˖ "MFBUPSJD6ODFSUBJOUZ 4UBUJTUJDBM6ODFSUBJOUZ

˖ &QJTUFNJD6ODFSUBJOUZ 4ZTUFNBUJDVODFSUBJOUZ

㼎韋植韋ח׉׮׉׮ろת׸׷劤颵涸ז⩐搫䚍ח״׷չ♶然㹋䚍պկずׄ㹋꿀װ鎘庠׾✳㔐װ׷
ה⦼ָת׏׋ֻずׄחכז׵זְկ➂䩛ד➰♷ׅ׷侄䌌ٓكٕךJODPOTJTUFODZזו׮ろ׬կ
״׶㢳ֻך䞔㜠׾꧊׭׋ה׃ג׮⡚幾ׅ׷ֿהָדֹזְ♶然ַׁկ
濼陎ָ♶駈֮׷ְכٌرָٕ♶姻然٥♶㸣Ⰻד֮׷ֿהח饯㔓ׅ׷չ♶然㹋䚍պկ植儗挿
דכⰅ䩛דֹזְ䞔㜠ָ֮׷׋׭ח欰ׄ׷♶然ַׁկ
⩐搫铎䊴PS⩐搫涸ז♶然㹋ׁ
禸窟铎䊴PS钠陎锷涸ז♶然㹋ׁ

View Slide

%JTUSJCVUJPO'SFF6ODFSUBJOUZ2VBOUJGJDBUJPO 62

• Accuracy alone does not suffice for reliable, consequential decision-making; we also need
uncertainty.
• Distribution-free UQ gives finite-sample statistical guarantees for any predictive model, no
matter how bad/misspecified, and any data distribution, even if unknown.
• DF techniques such as conformal prediction represent a new, principled approach to UQ for
complex prediction systems, such as deep learning.

View Slide

%JTUSJCVUJPO'SFF6ODFSUBJOUZ2VBOUJGJDBUJPO 62

• conformal prediction
• tolerance regions
• risk-controlling prediction sets
• calibration by binning
and more

View Slide

$POGPSNBMQSFEJDUJPO $1

https://en.wikipedia.org/wiki/Conformal_prediction

View Slide


"Conformity" 侭さ䚍
Non-conformity measure ♶侭さ䚍㛇彊
֮׷⢽ָ׉׸תדך⢽ה嫰ץגו׸ֻ׵ְVOVTVBMַ׾庠׶ծ$1،ٕ؞ٔؤيכ
ֿך♶侭さ䚍׾⯋ח✮庠⼒꟦׾圓眠ׅ׷

View Slide


https://github.com/valeman/awesome-conformal-prediction
Awesome Conformal Prediction

View Slide


https://www.stat.cmu.edu/~aramdas/conformal.html

View Slide

."1*&.PEFM"HOPTUJD1SFEJDUJPO*OUFSWBM&TUJNBUPS
https://github.com/scikit-learn-contrib/MAPIE

View Slide

With MAPIE, uncertainties are back in machine learning
https://towardsdatascience.com/with-mapie-uncertainties-are-back-in-machine-learning-882d5c17fdc3

View Slide

https://github.com/scikit-learn-contrib/MAPIE
regressor = ExtraTreesRegressor(max_leaf_nodes=32, bootstrap=True)
MapieRegressor(regressor,
method="plus", cv=-1)
method="plus",
cv=Subsample(n_resampling
s=50))
method="plus", cv=-1)
method="plus",
cv=Subsample(n_resampling
s=50))
95% prediction intervals
Jacknife+
Jacknife+ after bootstrap
Jacknife+
Jacknife+ after bootstrap

View Slide

決定森回帰の信頼区間推定, Benign Overfitting, 多変量木とReLUネットの入力空間分割

決定森回帰の信頼区間推定, Benign Overfitting, 多変量木とReLUネットの入力空間分割

itakigawa

More Decks by itakigawa

Other Decks in Research

Featured

Transcript