機械学習と解釈可能性 ソフトウェアジャパン 2019 38 Slides LINE株式会社 DataLabs Takahiro Yoshinaga

本⽇の内容 2 • ⾃⼰紹介 • Introduction • 機械学習モデルと結果を解釈する⽅法 • どの特徴量が重要か • 各特徴量が予測にどう影響するか • 予測に対して特徴量がどう寄与するか • Summary

3 ⾃⼰紹介

⾃⼰紹介 4 • 吉永 尊洸（よしなが たかひろ） ü @t_yoshinaga0106 • 所属 : LINE株式会社 DataLabs • 経歴 ü ~2015.03, Ph.D. : 東⼤（素粒⼦論：SUSY現象論） ü ~2018.01 : データ分析の専⾨会社 ü 2018.02~ : 現職 • 最近の趣味 ü 機械学習の解釈性、息⼦の強化学習

5 Introduction

Introduction 6 機械学習の解釈性の重要性が⾼まっている • 現在の機械学習 : ブラックボックスになりがち • ⾼精度の予測は得意だが根拠の説明は苦⼿ • 今後、導⼊を拡⼤していくためには、根拠が説明で きる（≒ ⼈間が解釈できる）ことが重要 • 社会的な背景 • サービス提供側に説明責任が求められている ü ⽇本 : AI利活⽤原則案（総務省, 2018） ü EU : ⼀般データ保護規則（GDPR） ü US : 説明可能AI（XAIプロジェクト）

Introduction 7 研究界でも解釈性の論⽂が増えている https://www.ai-gakkai.or.jp/my-bookmark_vol33-no3/ 近年の研究動向は、原先⽣のまとめ参照

本⽇の内容 8 実務的な観点で解釈性がどう活⽤できるか 1. どの特徴量が重要か p Feature Importance 2. 各特徴量が予測にどう影響するか p Partial Dependence p Surrogate Model : Appendix 3. ある予測結果に対して特徴量がどう寄与するか p LIME, SHAP p Grad-CAM, Grad-CAM++

9 機械学習モデルと 結果を解釈する⽅法

機械学習モデルと結果を解釈する⽅法 10 ⼤きくは、以下の3つの説明ができること 1. どの特徴量が重要か p Feature Importance 2. 各特徴量が予測にどう影響するか p Partial Dependence p Surrogate Model : Appendix 3. ある予測結果に対して特徴量がどう寄与するか p LIME, SHAP p Grad-CAM, Grad-CAM++ （モデルの説明 : 1, 2 結果の説明 : 3）

機械学習モデルと結果を解釈する⽅法 11 ⼤きくは、以下の3つの説明ができること 1. どの特徴量が重要か p モデルが重要視している要因がわかる 2. 各特徴量が予測にどう影響するか p 特徴量を変化させたときの予測の傾向がわかる 3. ある予測結果に対して特徴量がどう寄与するか p なぜその予測値だったか要因がわかる （モデルの説明 : 1, 2 結果の説明 : 3）

機械学習モデルと結果を解釈する⽅法 12 1. どの特徴量が重要か p Feature Importance 2. 各特徴量が予測にどう影響するか p Partial Dependence p Surrogate Model : Appendix 3. ある予測結果に対して特徴量がどう寄与するか p LIME, SHAP p Grad-CAM, Grad-CAM++ ⼤きくは、以下の3つの説明ができること （モデルの説明 : 1, 2 結果の説明 : 3）

Feature Importance 13 モデルに使⽤する特徴量の重要度を数値化したもの • Model dependent • RandomForest : 評価指標をより改善する特徴量が重要 ü 分類 : OOB（Out-of-Bag）の誤差率、ジニ係数 ü 回帰 : MSE, RSS • XGBoost ü gain, cover, weight (frequency) • … • Model independent • Permutation Importance ü 値をランダムにしたときに精度が落ちる = 重要度が⾼い モデルによって詳細が異なるので注意

Feature Importance : R実装 14 • 汎⽤的なものはmlrの標準関数として搭載 • mlr : Machine Learning in R ü Rの汎⽤型Machine Learningパッケージ (*) • データセット : ボストンの家賃価格 • ブラックボックスモデル : RandomForest https://mlr.mlr-org.com/ (*) 今後はmlrよりもtidymodelsのほうがスタンダードになるかもしれません

Feature Importance : R実装 15 • 学習の流れ • Task（データセットと⽬的変数）を定義 • Leaner（回帰/分類、機械学習モデル）を定義 • （ハイパーパラメータ探索：モデル依存） • TaskとLearnerを⽤いて学習 https://mlr.mlr-org.com/

Feature Importance : R実装 16 • Feature Importance • generateFeatureImportanceData() ü モデルに依存しない重要度を計算 重要度 より重要 重要な特徴量が わかる https://mlr.mlr-org.com/

機械学習モデルの結果を解釈する⽅法 17 1. どの特徴量が重要か p Feature Importance 2. 各特徴量が予測にどう影響するか p Partial Dependence p Surrogate Model : Appendix 3. ある予測結果に対して特徴量がどう寄与するか p LIME, SHAP p Grad-CAM, Grad-CAM++ ⼤きくは、以下の3つの説明ができること （モデルの説明 : 1, 2 結果の説明 : 3）

Partial Dependence 18 特徴量を変化させたときの出⼒の平均的な変化 ˆ f(xS) = ExC [ ˆ f(xS, xC)] = Z ˆ f(xS, xC)dP(xC) AAACuHichVFNSxtBGH5ctU1TW2N7KfSyGBQDRSalUAkUpKHgMbpGhWzY7q6TZHC/2J2ExCV/oH+gh55aEJH+gp578Q948ORV8Wihlx58d7Ogbai+y84887zv884zM1bgiEgydjqhTE5NP3iYe5R/PPPk6Wxh7tlW5HdDm9dt3/HDHcuMuCM8XpdCOnwnCLnpWg7ftvaqSX67x8NI+N6mHAS86ZptT7SEbUqijIKmd0wZt4ZLfUMrqe/UD0bcN6rDxi36lUpMqUlJXXhSHcuou7pryo5lxbWErZaMQpEtszTUcVDOQBFZ1PzCIXTswoeNLlxweJCEHZiI6GugDIaAuCZi4kJCIs1zDJEnbZeqOFWYxO7R2KZVI2M9Wic9o1Rt0y4O/SEpVSywE3bErtgx+84u2J//9orTHomXAc3WSMsDY/bTC+33vSqXZonOjepOzxItrKReBXkPUiY5hT3S9/Y/X2mVjYV4kX1jl+T/KztlP+kEXu+XfbDON74gTw9Q/ve6x8HW6+Uy4fU3xdX32VPk8BLzWKL7fotVrKGGOu37A2c4x4VSUT4qbUWMSpWJTPMcf4USXgO9sKg4 AAACuHichVFNSxtBGH5ctU1TW2N7KfSyGBQDRSalUAkUpKHgMbpGhWzY7q6TZHC/2J2ExCV/oH+gh55aEJH+gp578Q948ORV8Wihlx58d7Ogbai+y84887zv884zM1bgiEgydjqhTE5NP3iYe5R/PPPk6Wxh7tlW5HdDm9dt3/HDHcuMuCM8XpdCOnwnCLnpWg7ftvaqSX67x8NI+N6mHAS86ZptT7SEbUqijIKmd0wZt4ZLfUMrqe/UD0bcN6rDxi36lUpMqUlJXXhSHcuou7pryo5lxbWErZaMQpEtszTUcVDOQBFZ1PzCIXTswoeNLlxweJCEHZiI6GugDIaAuCZi4kJCIs1zDJEnbZeqOFWYxO7R2KZVI2M9Wic9o1Rt0y4O/SEpVSywE3bErtgx+84u2J//9orTHomXAc3WSMsDY/bTC+33vSqXZonOjepOzxItrKReBXkPUiY5hT3S9/Y/X2mVjYV4kX1jl+T/KztlP+kEXu+XfbDON74gTw9Q/ve6x8HW6+Uy4fU3xdX32VPk8BLzWKL7fotVrKGGOu37A2c4x4VSUT4qbUWMSpWJTPMcf4USXgO9sKg4 AAACuHichVFNSxtBGH5ctU1TW2N7KfSyGBQDRSalUAkUpKHgMbpGhWzY7q6TZHC/2J2ExCV/oH+gh55aEJH+gp578Q948ORV8Wihlx58d7Ogbai+y84887zv884zM1bgiEgydjqhTE5NP3iYe5R/PPPk6Wxh7tlW5HdDm9dt3/HDHcuMuCM8XpdCOnwnCLnpWg7ftvaqSX67x8NI+N6mHAS86ZptT7SEbUqijIKmd0wZt4ZLfUMrqe/UD0bcN6rDxi36lUpMqUlJXXhSHcuou7pryo5lxbWErZaMQpEtszTUcVDOQBFZ1PzCIXTswoeNLlxweJCEHZiI6GugDIaAuCZi4kJCIs1zDJEnbZeqOFWYxO7R2KZVI2M9Wic9o1Rt0y4O/SEpVSywE3bErtgx+84u2J//9orTHomXAc3WSMsDY/bTC+33vSqXZonOjepOzxItrKReBXkPUiY5hT3S9/Y/X2mVjYV4kX1jl+T/KztlP+kEXu+XfbDON74gTw9Q/ve6x8HW6+Uy4fU3xdX32VPk8BLzWKL7fotVrKGGOu37A2c4x4VSUT4qbUWMSpWJTPMcf4USXgO9sKg4 AAACuHichVFNSxtBGH5ctU1TW2N7KfSyGBQDRSalUAkUpKHgMbpGhWzY7q6TZHC/2J2ExCV/oH+gh55aEJH+gp578Q948ORV8Wihlx58d7Ogbai+y84887zv884zM1bgiEgydjqhTE5NP3iYe5R/PPPk6Wxh7tlW5HdDm9dt3/HDHcuMuCM8XpdCOnwnCLnpWg7ftvaqSX67x8NI+N6mHAS86ZptT7SEbUqijIKmd0wZt4ZLfUMrqe/UD0bcN6rDxi36lUpMqUlJXXhSHcuou7pryo5lxbWErZaMQpEtszTUcVDOQBFZ1PzCIXTswoeNLlxweJCEHZiI6GugDIaAuCZi4kJCIs1zDJEnbZeqOFWYxO7R2KZVI2M9Wic9o1Rt0y4O/SEpVSywE3bErtgx+84u2J//9orTHomXAc3WSMsDY/bTC+33vSqXZonOjepOzxItrKReBXkPUiY5hT3S9/Y/X2mVjYV4kX1jl+T/KztlP+kEXu+XfbDON74gTw9Q/ve6x8HW6+Uy4fU3xdX32VPk8BLzWKL7fotVrKGGOu37A2c4x4VSUT4qbUWMSpWJTPMcf4USXgO9sKg4 モデルの出⼒ 1. ある に対して xS AAACZnichVFNSwJBGH7cvsxKrYiCLpIYnWSUoOgkdOnoR36Aiexuoy3uF7urZNIfCLrmoVNBRPQzuvQHOvgPio4GXTr0ui5ESfUOM/PMM+/zzjMzkqkqtsNYzyeMjU9MTvmnAzOzc8FQeH6hYBtNS+Z52VANqySJNlcVnecdxVF5ybS4qEkqL0qN3cF+scUtWzH0fadt8oom1nWlpsiiQ1TuuJqrhqMsztyIjIKEB6LwIm2Eb3GAQxiQ0YQGDh0OYRUibGplJMBgEldBhziLkOLuc5wiQNomZXHKEIlt0FinVdljdVoPatquWqZTVOoWKSOIsSd2x/rskd2zF/bxa62OW2PgpU2zNNRysxo6W8m9/6vSaHZw9KX607ODGrZdrwp5N11mcAt5qG+ddPu5nWyss86u2Sv5v2I99kA30Ftv8k2GZy8RoA9I/HzuUVBIxhOEM5vRVNL7Cj9WsYYNeu8tpLCHNPJ0bh3nuEDX9ywEhSVheZgq+DzNIr6FEPkEoa2Krg== AAACZnichVFNSwJBGH7cvsxKrYiCLpIYnWSUoOgkdOnoR36Aiexuoy3uF7urZNIfCLrmoVNBRPQzuvQHOvgPio4GXTr0ui5ESfUOM/PMM+/zzjMzkqkqtsNYzyeMjU9MTvmnAzOzc8FQeH6hYBtNS+Z52VANqySJNlcVnecdxVF5ybS4qEkqL0qN3cF+scUtWzH0fadt8oom1nWlpsiiQ1TuuJqrhqMsztyIjIKEB6LwIm2Eb3GAQxiQ0YQGDh0OYRUibGplJMBgEldBhziLkOLuc5wiQNomZXHKEIlt0FinVdljdVoPatquWqZTVOoWKSOIsSd2x/rskd2zF/bxa62OW2PgpU2zNNRysxo6W8m9/6vSaHZw9KX607ODGrZdrwp5N11mcAt5qG+ddPu5nWyss86u2Sv5v2I99kA30Ftv8k2GZy8RoA9I/HzuUVBIxhOEM5vRVNL7Cj9WsYYNeu8tpLCHNPJ0bh3nuEDX9ywEhSVheZgq+DzNIr6FEPkEoa2Krg== AAACZnichVFNSwJBGH7cvsxKrYiCLpIYnWSUoOgkdOnoR36Aiexuoy3uF7urZNIfCLrmoVNBRPQzuvQHOvgPio4GXTr0ui5ESfUOM/PMM+/zzjMzkqkqtsNYzyeMjU9MTvmnAzOzc8FQeH6hYBtNS+Z52VANqySJNlcVnecdxVF5ybS4qEkqL0qN3cF+scUtWzH0fadt8oom1nWlpsiiQ1TuuJqrhqMsztyIjIKEB6LwIm2Eb3GAQxiQ0YQGDh0OYRUibGplJMBgEldBhziLkOLuc5wiQNomZXHKEIlt0FinVdljdVoPatquWqZTVOoWKSOIsSd2x/rskd2zF/bxa62OW2PgpU2zNNRysxo6W8m9/6vSaHZw9KX607ODGrZdrwp5N11mcAt5qG+ddPu5nWyss86u2Sv5v2I99kA30Ftv8k2GZy8RoA9I/HzuUVBIxhOEM5vRVNL7Cj9WsYYNeu8tpLCHNPJ0bh3nuEDX9ywEhSVheZgq+DzNIr6FEPkEoa2Krg== AAACZnichVFNSwJBGH7cvsxKrYiCLpIYnWSUoOgkdOnoR36Aiexuoy3uF7urZNIfCLrmoVNBRPQzuvQHOvgPio4GXTr0ui5ESfUOM/PMM+/zzjMzkqkqtsNYzyeMjU9MTvmnAzOzc8FQeH6hYBtNS+Z52VANqySJNlcVnecdxVF5ybS4qEkqL0qN3cF+scUtWzH0fadt8oom1nWlpsiiQ1TuuJqrhqMsztyIjIKEB6LwIm2Eb3GAQxiQ0YQGDh0OYRUibGplJMBgEldBhziLkOLuc5wiQNomZXHKEIlt0FinVdljdVoPatquWqZTVOoWKSOIsSd2x/rskd2zF/bxa62OW2PgpU2zNNRysxo6W8m9/6vSaHZw9KX607ODGrZdrwp5N11mcAt5qG+ddPu5nWyss86u2Sv5v2I99kA30Ftv8k2GZy8RoA9I/HzuUVBIxhOEM5vRVNL7Cj9WsYYNeu8tpLCHNPJ0bh3nuEDX9ywEhSVheZgq+DzNIr6FEPkEoa2Krg== xC AAACZnichVHLSsNAFD2Nr1ofrYoouCmWiqsyKYLiqtCNyz7sA2opSZzW0DQJSVqsxR8Q3NqFKwUR8TPc+AMu+geKywpuXHibBkSLeoeZOXPmnjtnZmRTU22HsZ5PGBufmJzyTwdmZufmg6GFxbxtNC2F5xRDM6yiLNlcU3Wec1RH40XT4lJD1nhBricH+4UWt2zV0PedtsnLDammq1VVkRyisseVZCUUYTHmRngUiB6IwIuUEbrFAQ5hQEETDXDocAhrkGBTK0EEg0lcGR3iLEKqu89xigBpm5TFKUMitk5jjVYlj9VpPahpu2qFTtGoW6QMI8qe2B3rs0d2z17Yx6+1Om6NgZc2zfJQy81K8Gw1+/6vqkGzg6Mv1Z+eHVSx43pVybvpMoNbKEN966Tbz+5mop0Nds1eyf8V67EHuoHeelNu0jxziQB9gPjzuUdBPh4TCae3Iom49xV+rGEdm/Te20hgDynk6NwaznGBru9ZmBeWhZVhquDzNEv4FkL4E4Gtip4= AAACZnichVHLSsNAFD2Nr1ofrYoouCmWiqsyKYLiqtCNyz7sA2opSZzW0DQJSVqsxR8Q3NqFKwUR8TPc+AMu+geKywpuXHibBkSLeoeZOXPmnjtnZmRTU22HsZ5PGBufmJzyTwdmZufmg6GFxbxtNC2F5xRDM6yiLNlcU3Wec1RH40XT4lJD1nhBricH+4UWt2zV0PedtsnLDammq1VVkRyisseVZCUUYTHmRngUiB6IwIuUEbrFAQ5hQEETDXDocAhrkGBTK0EEg0lcGR3iLEKqu89xigBpm5TFKUMitk5jjVYlj9VpPahpu2qFTtGoW6QMI8qe2B3rs0d2z17Yx6+1Om6NgZc2zfJQy81K8Gw1+/6vqkGzg6Mv1Z+eHVSx43pVybvpMoNbKEN966Tbz+5mop0Nds1eyf8V67EHuoHeelNu0jxziQB9gPjzuUdBPh4TCae3Iom49xV+rGEdm/Te20hgDynk6NwaznGBru9ZmBeWhZVhquDzNEv4FkL4E4Gtip4= AAACZnichVHLSsNAFD2Nr1ofrYoouCmWiqsyKYLiqtCNyz7sA2opSZzW0DQJSVqsxR8Q3NqFKwUR8TPc+AMu+geKywpuXHibBkSLeoeZOXPmnjtnZmRTU22HsZ5PGBufmJzyTwdmZufmg6GFxbxtNC2F5xRDM6yiLNlcU3Wec1RH40XT4lJD1nhBricH+4UWt2zV0PedtsnLDammq1VVkRyisseVZCUUYTHmRngUiB6IwIuUEbrFAQ5hQEETDXDocAhrkGBTK0EEg0lcGR3iLEKqu89xigBpm5TFKUMitk5jjVYlj9VpPahpu2qFTtGoW6QMI8qe2B3rs0d2z17Yx6+1Om6NgZc2zfJQy81K8Gw1+/6vqkGzg6Mv1Z+eHVSx43pVybvpMoNbKEN966Tbz+5mop0Nds1eyf8V67EHuoHeelNu0jxziQB9gPjzuUdBPh4TCae3Iom49xV+rGEdm/Te20hgDynk6NwaznGBru9ZmBeWhZVhquDzNEv4FkL4E4Gtip4= AAACZnichVHLSsNAFD2Nr1ofrYoouCmWiqsyKYLiqtCNyz7sA2opSZzW0DQJSVqsxR8Q3NqFKwUR8TPc+AMu+geKywpuXHibBkSLeoeZOXPmnjtnZmRTU22HsZ5PGBufmJzyTwdmZufmg6GFxbxtNC2F5xRDM6yiLNlcU3Wec1RH40XT4lJD1nhBricH+4UWt2zV0PedtsnLDammq1VVkRyisseVZCUUYTHmRngUiB6IwIuUEbrFAQ5hQEETDXDocAhrkGBTK0EEg0lcGR3iLEKqu89xigBpm5TFKUMitk5jjVYlj9VpPahpu2qFTtGoW6QMI8qe2B3rs0d2z17Yx6+1Om6NgZc2zfJQy81K8Gw1+/6vqkGzg6Mv1Z+eHVSx43pVybvpMoNbKEN966Tbz+5mop0Nds1eyf8V67EHuoHeelNu0jxziQB9gPjzuUdBPh4TCae3Iom49xV+rGEdm/Te20hgDynk6NwaznGBru9ZmBeWhZVhquDzNEv4FkL4E4Gtip4= 2. で周辺化 xS xC1 xC2 xC3 0.22 4 5 3 0.31 3 6 5 0.44 3 7 1 元データ xS xC1 xC2 xC3 0.22 4 5 3 0.22 3 6 5 0.22 3 7 1 xS xC1 xC2 xC3 0.44 4 5 3 0.44 3 6 5 0.44 3 7 1 xS xC1 xC2 xC3 0.31 4 5 3 0.31 3 6 5 0.31 3 7 1 f(0.22) f(0.31) f(0.44) 出⼒をxS以外で平均

Partial Dependence : R実装 19 • Partial Dependence Plot • generatePartialDependenceData()に 学習済モデルとtaskとFeatureの名前を⼊⼒ • plotPartialDependence()でplot RandomForest 重要度が⾼い順 平均以外も選択可能 https://mlr.mlr-org.com/

Partial Dependence : R実装 20 • Partial Dependence Plot • 横軸：特徴量 • 縦軸：特徴量を動かしたときの出⼒の平均的な動き https://mlr.mlr-org.com/

Partial Dependence : R実装 21 • Partial Dependence Plot • 横軸：特徴量 • 縦軸：特徴量を動かしたときの出⼒の平均的な動き 重要なだけでなく どう重要かがわかる(*) https://mlr.mlr-org.com/ （*）特徴量間の相関が強い場合、結果は信頼できない ( cf) Accumulated Local Effect)

機械学習モデルの結果を解釈する⽅法 22 1. どの特徴量が重要か p Feature Importance 2. 各特徴量が予測にどう影響するか p Partial Dependence p Surrogate Model : Appendix 3. ある予測結果に対して特徴量がどう寄与するか p LIME, SHAP p Grad-CAM, Grad-CAM++ ⼤きくは、以下の3つの説明ができること （モデルの説明 : 1, 2 結果の説明 : 3）

LIME, SHAP 23 説明したいデータ付近でモデルを近似する [Rebeiro+, 16] 局所的な分類器 モデルが負と 判定する領域 モデルが正と 判定する領域 正例 負例 説明したい データ

LIME, SHAP 24 説明したいデータ付近でモデルを近似する [Lundberg, Lee, 17] x AAACZHichVHLSsNAFD2Nr1qrrRZBEKRYKq5kIoLiqujGZR/2AbWUJE5raJqEJC3W4g/oVnHhSkFE/Aw3/oCL/oAgLiu4ceFtGhAt6h1m5syZe+6cmZFNTbUdxjo+YWh4ZHTMPx6YCE5OhcLTMznbaFgKzyqGZlgFWbK5puo866iOxgumxaW6rPG8XNvu7eeb3LJVQ991WiYv1aWqrlZURXKISh2WwzG2wtyIDgLRAzF4kTTCt9jDPgwoaKAODh0OYQ0SbGpFiGAwiSuhTZxFSHX3OY4RIG2DsjhlSMTWaKzSquixOq17NW1XrdApGnWLlFHE2RO7Y132yO7ZC/v4tVbbrdHz0qJZ7mu5WQ6dzGXe/1XVaXZw8KX607ODCjZcryp5N12mdwulr28eXXQzm+l4e4lds1fyf8U67IFuoDfflJsUT18iQB8g/nzuQZBbXREJp9ZiiS3vK/yYxyKW6b3XkcAOksjSuRynOMO571kIChFhtp8q+DxNBN9CWPgE9k6J+A== AAACZHichVHLSsNAFD2Nr1qrrRZBEKRYKq5kIoLiqujGZR/2AbWUJE5raJqEJC3W4g/oVnHhSkFE/Aw3/oCL/oAgLiu4ceFtGhAt6h1m5syZe+6cmZFNTbUdxjo+YWh4ZHTMPx6YCE5OhcLTMznbaFgKzyqGZlgFWbK5puo866iOxgumxaW6rPG8XNvu7eeb3LJVQ991WiYv1aWqrlZURXKISh2WwzG2wtyIDgLRAzF4kTTCt9jDPgwoaKAODh0OYQ0SbGpFiGAwiSuhTZxFSHX3OY4RIG2DsjhlSMTWaKzSquixOq17NW1XrdApGnWLlFHE2RO7Y132yO7ZC/v4tVbbrdHz0qJZ7mu5WQ6dzGXe/1XVaXZw8KX607ODCjZcryp5N12mdwulr28eXXQzm+l4e4lds1fyf8U67IFuoDfflJsUT18iQB8g/nzuQZBbXREJp9ZiiS3vK/yYxyKW6b3XkcAOksjSuRynOMO571kIChFhtp8q+DxNBN9CWPgE9k6J+A== AAACZHichVHLSsNAFD2Nr1qrrRZBEKRYKq5kIoLiqujGZR/2AbWUJE5raJqEJC3W4g/oVnHhSkFE/Aw3/oCL/oAgLiu4ceFtGhAt6h1m5syZe+6cmZFNTbUdxjo+YWh4ZHTMPx6YCE5OhcLTMznbaFgKzyqGZlgFWbK5puo866iOxgumxaW6rPG8XNvu7eeb3LJVQ991WiYv1aWqrlZURXKISh2WwzG2wtyIDgLRAzF4kTTCt9jDPgwoaKAODh0OYQ0SbGpFiGAwiSuhTZxFSHX3OY4RIG2DsjhlSMTWaKzSquixOq17NW1XrdApGnWLlFHE2RO7Y132yO7ZC/v4tVbbrdHz0qJZ7mu5WQ6dzGXe/1XVaXZw8KX607ODCjZcryp5N12mdwulr28eXXQzm+l4e4lds1fyf8U67IFuoDfflJsUT18iQB8g/nzuQZBbXREJp9ZiiS3vK/yYxyKW6b3XkcAOksjSuRynOMO571kIChFhtp8q+DxNBN9CWPgE9k6J+A== AAACZHichVHLSsNAFD2Nr1qrrRZBEKRYKq5kIoLiqujGZR/2AbWUJE5raJqEJC3W4g/oVnHhSkFE/Aw3/oCL/oAgLiu4ceFtGhAt6h1m5syZe+6cmZFNTbUdxjo+YWh4ZHTMPx6YCE5OhcLTMznbaFgKzyqGZlgFWbK5puo866iOxgumxaW6rPG8XNvu7eeb3LJVQ991WiYv1aWqrlZURXKISh2WwzG2wtyIDgLRAzF4kTTCt9jDPgwoaKAODh0OYQ0SbGpFiGAwiSuhTZxFSHX3OY4RIG2DsjhlSMTWaKzSquixOq17NW1XrdApGnWLlFHE2RO7Y132yO7ZC/v4tVbbrdHz0qJZ7mu5WQ6dzGXe/1XVaXZw8KX607ODCjZcryp5N12mdwulr28eXXQzm+l4e4lds1fyf8U67IFuoDfflJsUT18iQB8g/nzuQZBbXREJp9ZiiS3vK/yYxyKW6b3XkcAOksjSuRynOMO571kIChFhtp8q+DxNBN9CWPgE9k6J+A== x0 AAACbXichVHLSgMxFD0d3/XRqgiCImLxsSoZERRXohuXVq2KtsrMmNbgvJhJi1r8AdeCC1FQEBE/w40/4MJPEBcuKrhx4Z3pgKioNyQ5Obnn5iTRXVP4krHHmFJX39DY1NwSb21r70gkO7tWfKfkGTxrOKbjremaz01h86wU0uRrrsc1Szf5qr47F+yvlrnnC8delvsuz1ta0RYFYWiSqPW9zUrO9YTFD7eSKZZmYQz+BGoEUohiwUleI4dtODBQggUOG5KwCQ0+tQ2oYHCJy6NCnEdIhPsch4iTtkRZnDI0YndpLNJqI2JtWgc1/VBt0CkmdY+UgxhmD+yGVdk9u2VP7P3XWpWwRuBln2a9puXuVuKod+ntX5VFs8TOp+pPzxIFTIVeBXl3Qya4hVHTlw9OqkvTi8OVEXbJnsn/BXtkd3QDu/xqXGX44ini9AHq9+f+CVbG0yrhzERqZjb6imb0YQhj9N6TmME8FpClc20c4wznsRelR+lXBmqpSizSdONLKKMf+JyOIQ== AAACbXichVHLSgMxFD0d3/XRqgiCImLxsSoZERRXohuXVq2KtsrMmNbgvJhJi1r8AdeCC1FQEBE/w40/4MJPEBcuKrhx4Z3pgKioNyQ5Obnn5iTRXVP4krHHmFJX39DY1NwSb21r70gkO7tWfKfkGTxrOKbjremaz01h86wU0uRrrsc1Szf5qr47F+yvlrnnC8delvsuz1ta0RYFYWiSqPW9zUrO9YTFD7eSKZZmYQz+BGoEUohiwUleI4dtODBQggUOG5KwCQ0+tQ2oYHCJy6NCnEdIhPsch4iTtkRZnDI0YndpLNJqI2JtWgc1/VBt0CkmdY+UgxhmD+yGVdk9u2VP7P3XWpWwRuBln2a9puXuVuKod+ntX5VFs8TOp+pPzxIFTIVeBXl3Qya4hVHTlw9OqkvTi8OVEXbJnsn/BXtkd3QDu/xqXGX44ini9AHq9+f+CVbG0yrhzERqZjb6imb0YQhj9N6TmME8FpClc20c4wznsRelR+lXBmqpSizSdONLKKMf+JyOIQ== AAACbXichVHLSgMxFD0d3/XRqgiCImLxsSoZERRXohuXVq2KtsrMmNbgvJhJi1r8AdeCC1FQEBE/w40/4MJPEBcuKrhx4Z3pgKioNyQ5Obnn5iTRXVP4krHHmFJX39DY1NwSb21r70gkO7tWfKfkGTxrOKbjremaz01h86wU0uRrrsc1Szf5qr47F+yvlrnnC8delvsuz1ta0RYFYWiSqPW9zUrO9YTFD7eSKZZmYQz+BGoEUohiwUleI4dtODBQggUOG5KwCQ0+tQ2oYHCJy6NCnEdIhPsch4iTtkRZnDI0YndpLNJqI2JtWgc1/VBt0CkmdY+UgxhmD+yGVdk9u2VP7P3XWpWwRuBln2a9puXuVuKod+ntX5VFs8TOp+pPzxIFTIVeBXl3Qya4hVHTlw9OqkvTi8OVEXbJnsn/BXtkd3QDu/xqXGX44ini9AHq9+f+CVbG0yrhzERqZjb6imb0YQhj9N6TmME8FpClc20c4wznsRelR+lXBmqpSizSdONLKKMf+JyOIQ== AAACbXichVHLSgMxFD0d3/XRqgiCImLxsSoZERRXohuXVq2KtsrMmNbgvJhJi1r8AdeCC1FQEBE/w40/4MJPEBcuKrhx4Z3pgKioNyQ5Obnn5iTRXVP4krHHmFJX39DY1NwSb21r70gkO7tWfKfkGTxrOKbjremaz01h86wU0uRrrsc1Szf5qr47F+yvlrnnC8delvsuz1ta0RYFYWiSqPW9zUrO9YTFD7eSKZZmYQz+BGoEUohiwUleI4dtODBQggUOG5KwCQ0+tQ2oYHCJy6NCnEdIhPsch4iTtkRZnDI0YndpLNJqI2JtWgc1/VBt0CkmdY+UgxhmD+yGVdk9u2VP7P3XWpWwRuBln2a9puXuVuKod+ntX5VFs8TOp+pPzxIFTIVeBXl3Qya4hVHTlw9OqkvTi8OVEXbJnsn/BXtkd3QDu/xqXGX44ini9AHq9+f+CVbG0yrhzERqZjb6imb0YQhj9N6TmME8FpClc20c4wznsRelR+lXBmqpSizSdONLKKMf+JyOIQ== f(x) AAACZ3ichVHLSsNAFD2Nr1ofrQoiuFGLoptyK4LiSnTjsrZWBRVJ4tSGpklIpsVa/AEXbiu4UhARP8ONP+DCTyguK7hx4U0aEBX1DjNz5sw9d87MaI5peJLoOaJ0dHZ190R7Y339A4PxxNDwlmdXXF3kddu03R1N9YRpWCIvDWmKHccValkzxbZWWvP3t6vC9Qzb2pQ1R+yX1SPLKBi6Kn2qMHs8d5BIUoqCmPgJ0iFIIoyMnbjFHg5hQ0cFZQhYkIxNqPC47SINgsPcPurMuYyMYF/gFDHWVjhLcIbKbInHI17thqzFa7+mF6h1PsXk7rJyAtP0RHfUoke6pya9/1qrHtTwvdR41tpa4RzEz8Zyb/+qyjxLFD9Vf3qWKGAp8Gqwdydg/FvobX31pNHKLWen6zN0TS/s/4qe6YFvYFVf9ZsNkb1EjD8g/f25f4Kt+VSa8cZCcmU1/IooxjGFWX7vRaxgHRnk+dwiztHARaSpxJVRZaydqkRCzQi+hDL5AdaMis0= AAACZ3ichVHLSsNAFD2Nr1ofrQoiuFGLoptyK4LiSnTjsrZWBRVJ4tSGpklIpsVa/AEXbiu4UhARP8ONP+DCTyguK7hx4U0aEBX1DjNz5sw9d87MaI5peJLoOaJ0dHZ190R7Y339A4PxxNDwlmdXXF3kddu03R1N9YRpWCIvDWmKHccValkzxbZWWvP3t6vC9Qzb2pQ1R+yX1SPLKBi6Kn2qMHs8d5BIUoqCmPgJ0iFIIoyMnbjFHg5hQ0cFZQhYkIxNqPC47SINgsPcPurMuYyMYF/gFDHWVjhLcIbKbInHI17thqzFa7+mF6h1PsXk7rJyAtP0RHfUoke6pya9/1qrHtTwvdR41tpa4RzEz8Zyb/+qyjxLFD9Vf3qWKGAp8Gqwdydg/FvobX31pNHKLWen6zN0TS/s/4qe6YFvYFVf9ZsNkb1EjD8g/f25f4Kt+VSa8cZCcmU1/IooxjGFWX7vRaxgHRnk+dwiztHARaSpxJVRZaydqkRCzQi+hDL5AdaMis0= AAACZ3ichVHLSsNAFD2Nr1ofrQoiuFGLoptyK4LiSnTjsrZWBRVJ4tSGpklIpsVa/AEXbiu4UhARP8ONP+DCTyguK7hx4U0aEBX1DjNz5sw9d87MaI5peJLoOaJ0dHZ190R7Y339A4PxxNDwlmdXXF3kddu03R1N9YRpWCIvDWmKHccValkzxbZWWvP3t6vC9Qzb2pQ1R+yX1SPLKBi6Kn2qMHs8d5BIUoqCmPgJ0iFIIoyMnbjFHg5hQ0cFZQhYkIxNqPC47SINgsPcPurMuYyMYF/gFDHWVjhLcIbKbInHI17thqzFa7+mF6h1PsXk7rJyAtP0RHfUoke6pya9/1qrHtTwvdR41tpa4RzEz8Zyb/+qyjxLFD9Vf3qWKGAp8Gqwdydg/FvobX31pNHKLWen6zN0TS/s/4qe6YFvYFVf9ZsNkb1EjD8g/f25f4Kt+VSa8cZCcmU1/IooxjGFWX7vRaxgHRnk+dwiztHARaSpxJVRZaydqkRCzQi+hDL5AdaMis0= AAACZ3ichVHLSsNAFD2Nr1ofrQoiuFGLoptyK4LiSnTjsrZWBRVJ4tSGpklIpsVa/AEXbiu4UhARP8ONP+DCTyguK7hx4U0aEBX1DjNz5sw9d87MaI5peJLoOaJ0dHZ190R7Y339A4PxxNDwlmdXXF3kddu03R1N9YRpWCIvDWmKHccValkzxbZWWvP3t6vC9Qzb2pQ1R+yX1SPLKBi6Kn2qMHs8d5BIUoqCmPgJ0iFIIoyMnbjFHg5hQ0cFZQhYkIxNqPC47SINgsPcPurMuYyMYF/gFDHWVjhLcIbKbInHI17thqzFa7+mF6h1PsXk7rJyAtP0RHfUoke6pya9/1qrHtTwvdR41tpa4RzEz8Zyb/+qyjxLFD9Vf3qWKGAp8Gqwdydg/FvobX31pNHKLWen6zN0TS/s/4qe6YFvYFVf9ZsNkb1EjD8g/f25f4Kt+VSa8cZCcmU1/IooxjGFWX7vRaxgHRnk+dwiztHARaSpxJVRZaydqkRCzQi+hDL5AdaMis0= g(x0) AAACcHichVFNLwNBGH66vqo+WlwkDkpDcJCpSIiTcHFsS5Foye4aNel+2d02auMP+AMOLioRET/DxR9w8BPETSUuDt7dbiI0eCcz88wz7/POMzOKpQnHZewpIrW1d3R2RbtjPb19/fHEwOCmY1ZsledVUzPtbUV2uCYMnneFq/Fty+ayrmh8Symv+vtbVW47wjQ23JrFi7pcMsSBUGWXqGJp6njXK1i20Pnp9F4ixWZZEMlWkA5BCmFkzMQNCtiHCRUV6OAw4BLWIMOhtoM0GCziivCIswmJYJ/jFDHSViiLU4ZMbJnGEq12QtagtV/TCdQqnaJRt0mZxAR7ZLeswR7YHXtmH7/W8oIavpcazUpTy629+Nnw+vu/Kp1mF4dfqj89uzjAYuBVkHcrYPxbqE199eS8sb6Um/Am2RV7If919sTu6QZG9U29zvLcBWL0Aemfz90KNudm04Sz86nllfArohjBOKbovRewjDVkkKdzj3COS9Qjr9KwNCqNNVOlSKgZwreQZj4B4+mO9w== AAACcHichVFNLwNBGH66vqo+WlwkDkpDcJCpSIiTcHFsS5Foye4aNel+2d02auMP+AMOLioRET/DxR9w8BPETSUuDt7dbiI0eCcz88wz7/POMzOKpQnHZewpIrW1d3R2RbtjPb19/fHEwOCmY1ZsledVUzPtbUV2uCYMnneFq/Fty+ayrmh8Symv+vtbVW47wjQ23JrFi7pcMsSBUGWXqGJp6njXK1i20Pnp9F4ixWZZEMlWkA5BCmFkzMQNCtiHCRUV6OAw4BLWIMOhtoM0GCziivCIswmJYJ/jFDHSViiLU4ZMbJnGEq12QtagtV/TCdQqnaJRt0mZxAR7ZLeswR7YHXtmH7/W8oIavpcazUpTy629+Nnw+vu/Kp1mF4dfqj89uzjAYuBVkHcrYPxbqE199eS8sb6Um/Am2RV7If919sTu6QZG9U29zvLcBWL0Aemfz90KNudm04Sz86nllfArohjBOKbovRewjDVkkKdzj3COS9Qjr9KwNCqNNVOlSKgZwreQZj4B4+mO9w== AAACcHichVFNLwNBGH66vqo+WlwkDkpDcJCpSIiTcHFsS5Foye4aNel+2d02auMP+AMOLioRET/DxR9w8BPETSUuDt7dbiI0eCcz88wz7/POMzOKpQnHZewpIrW1d3R2RbtjPb19/fHEwOCmY1ZsledVUzPtbUV2uCYMnneFq/Fty+ayrmh8Symv+vtbVW47wjQ23JrFi7pcMsSBUGWXqGJp6njXK1i20Pnp9F4ixWZZEMlWkA5BCmFkzMQNCtiHCRUV6OAw4BLWIMOhtoM0GCziivCIswmJYJ/jFDHSViiLU4ZMbJnGEq12QtagtV/TCdQqnaJRt0mZxAR7ZLeswR7YHXtmH7/W8oIavpcazUpTy629+Nnw+vu/Kp1mF4dfqj89uzjAYuBVkHcrYPxbqE199eS8sb6Um/Am2RV7If919sTu6QZG9U29zvLcBWL0Aemfz90KNudm04Sz86nllfArohjBOKbovRewjDVkkKdzj3COS9Qjr9KwNCqNNVOlSKgZwreQZj4B4+mO9w== AAACcHichVFNLwNBGH66vqo+WlwkDkpDcJCpSIiTcHFsS5Foye4aNel+2d02auMP+AMOLioRET/DxR9w8BPETSUuDt7dbiI0eCcz88wz7/POMzOKpQnHZewpIrW1d3R2RbtjPb19/fHEwOCmY1ZsledVUzPtbUV2uCYMnneFq/Fty+ayrmh8Symv+vtbVW47wjQ23JrFi7pcMsSBUGWXqGJp6njXK1i20Pnp9F4ixWZZEMlWkA5BCmFkzMQNCtiHCRUV6OAw4BLWIMOhtoM0GCziivCIswmJYJ/jFDHSViiLU4ZMbJnGEq12QtagtV/TCdQqnaJRt0mZxAR7ZLeswR7YHXtmH7/W8oIavpcazUpTy629+Nnw+vu/Kp1mF4dfqj89uzjAYuBVkHcrYPxbqE199eS8sb6Um/Am2RV7If919sTu6QZG9U29zvLcBWL0Aemfz90KNudm04Sz86nllfArohjBOKbovRewjDVkkKdzj3COS9Qjr9KwNCqNNVOlSKgZwreQZj4B4+mO9w== b(x) AAACZ3ichVHLSsNAFD2N7/poVZCCG7Uouim3IiiuRDcuq7WtoEWSONqhaRKStFiLP+DCrYIrBRHxM9z4Ay78hOJSwY0Lb9KAaFHvMDNnztxz58yMZhvS9YieI0pHZ1d3T29ftH9gcCgWHx7Ju1bV0UVOtwzL2dZUVxjSFDlPeobYth2hVjRDFLTymr9fqAnHlZa55dVtUayoh6Y8kLrq+ZQ2ezS3F09SioKYaAfpECQRRsaK32IX+7Cgo4oKBEx4jA2ocLntIA2CzVwRDeYcRjLYFzhBlLVVzhKcoTJb5vGQVzsha/Lar+kGap1PMbg7rJzAND3RHb3SI91Tkz5+rdUIavhe6jxrLa2w92Kniez7v6oKzx5KX6o/PXs4wFLgVbJ3O2D8W+gtfe34/DW7vDndmKFremH/V/RMD3wDs/am32yIzUtE+QPSP5+7HeTnU2nGGwvJldXwK3oxjinM8nsvYgXryCDH55ZwhnNcRJpKTBlTEq1UJRJqRvEtlMlPzoCKyQ== AAACZ3ichVHLSsNAFD2N7/poVZCCG7Uouim3IiiuRDcuq7WtoEWSONqhaRKStFiLP+DCrYIrBRHxM9z4Ay78hOJSwY0Lb9KAaFHvMDNnztxz58yMZhvS9YieI0pHZ1d3T29ftH9gcCgWHx7Ju1bV0UVOtwzL2dZUVxjSFDlPeobYth2hVjRDFLTymr9fqAnHlZa55dVtUayoh6Y8kLrq+ZQ2ezS3F09SioKYaAfpECQRRsaK32IX+7Cgo4oKBEx4jA2ocLntIA2CzVwRDeYcRjLYFzhBlLVVzhKcoTJb5vGQVzsha/Lar+kGap1PMbg7rJzAND3RHb3SI91Tkz5+rdUIavhe6jxrLa2w92Kniez7v6oKzx5KX6o/PXs4wFLgVbJ3O2D8W+gtfe34/DW7vDndmKFremH/V/RMD3wDs/am32yIzUtE+QPSP5+7HeTnU2nGGwvJldXwK3oxjinM8nsvYgXryCDH55ZwhnNcRJpKTBlTEq1UJRJqRvEtlMlPzoCKyQ== AAACZ3ichVHLSsNAFD2N7/poVZCCG7Uouim3IiiuRDcuq7WtoEWSONqhaRKStFiLP+DCrYIrBRHxM9z4Ay78hOJSwY0Lb9KAaFHvMDNnztxz58yMZhvS9YieI0pHZ1d3T29ftH9gcCgWHx7Ju1bV0UVOtwzL2dZUVxjSFDlPeobYth2hVjRDFLTymr9fqAnHlZa55dVtUayoh6Y8kLrq+ZQ2ezS3F09SioKYaAfpECQRRsaK32IX+7Cgo4oKBEx4jA2ocLntIA2CzVwRDeYcRjLYFzhBlLVVzhKcoTJb5vGQVzsha/Lar+kGap1PMbg7rJzAND3RHb3SI91Tkz5+rdUIavhe6jxrLa2w92Kniez7v6oKzx5KX6o/PXs4wFLgVbJ3O2D8W+gtfe34/DW7vDndmKFremH/V/RMD3wDs/am32yIzUtE+QPSP5+7HeTnU2nGGwvJldXwK3oxjinM8nsvYgXryCDH55ZwhnNcRJpKTBlTEq1UJRJqRvEtlMlPzoCKyQ== AAACZ3ichVHLSsNAFD2N7/poVZCCG7Uouim3IiiuRDcuq7WtoEWSONqhaRKStFiLP+DCrYIrBRHxM9z4Ay78hOJSwY0Lb9KAaFHvMDNnztxz58yMZhvS9YieI0pHZ1d3T29ftH9gcCgWHx7Ju1bV0UVOtwzL2dZUVxjSFDlPeobYth2hVjRDFLTymr9fqAnHlZa55dVtUayoh6Y8kLrq+ZQ2ezS3F09SioKYaAfpECQRRsaK32IX+7Cgo4oKBEx4jA2ocLntIA2CzVwRDeYcRjLYFzhBlLVVzhKcoTJb5vGQVzsha/Lar+kGap1PMbg7rJzAND3RHb3SI91Tkz5+rdUIavhe6jxrLa2w92Kniez7v6oKzx5KX6o/PXs4wFLgVbJ3O2D8W+gtfe34/DW7vDndmKFremH/V/RMD3wDs/am32yIzUtE+QPSP5+7HeTnU2nGGwvJldXwK3oxjinM8nsvYgXryCDH55ZwhnNcRJpKTBlTEq1UJRJqRvEtlMlPzoCKyQ== hx(x0) AAACdHichVG7SgNBFD1Z3/GRqI2gRXCJaBMmIihWQRtLE00UfCy762gG98XuJBiDP+APWKRSFBE/w8YfsPATxDKijYU3mwVRUe8yO2fO3HPnzFzDs0QgGXuMKR2dXd09vX3x/oHBoURyeKQUuBXf5EXTtVx/09ADbgmHF6WQFt/0fK7bhsU3jMPl1v5GlfuBcJ11WfP4jq0fOGJfmLokSksmytrR9NFufdvzhc1PZrSkyjIsjNRPkI2AiihW3eQ1trEHFyYqsMHhQBK2oCOgbwtZMHjE7aBOnE9IhPscJ4iTtkJZnDJ0Yg/pf0CrrYh1aN2qGYRqk06xaPikTCHNHtgNa7J7dsue2PuvtephjZaXGs1GW8s9LXE6tvb2r8qmWaL8qfrTs8Q+FkKvgrx7IdO6hdnWV4/PmmuLhXR9il2wZ/J/zh7ZHd3Aqb6YV3leaCBODch+f+6foDSbyRLOz6m5pagVvRjHJKbpveeRwwpWUQx70sAlrmKvyoSiKul2qhKLNKP4EkrmA3EQkBQ= AAACdHichVG7SgNBFD1Z3/GRqI2gRXCJaBMmIihWQRtLE00UfCy762gG98XuJBiDP+APWKRSFBE/w8YfsPATxDKijYU3mwVRUe8yO2fO3HPnzFzDs0QgGXuMKR2dXd09vX3x/oHBoURyeKQUuBXf5EXTtVx/09ADbgmHF6WQFt/0fK7bhsU3jMPl1v5GlfuBcJ11WfP4jq0fOGJfmLokSksmytrR9NFufdvzhc1PZrSkyjIsjNRPkI2AiihW3eQ1trEHFyYqsMHhQBK2oCOgbwtZMHjE7aBOnE9IhPscJ4iTtkJZnDJ0Yg/pf0CrrYh1aN2qGYRqk06xaPikTCHNHtgNa7J7dsue2PuvtephjZaXGs1GW8s9LXE6tvb2r8qmWaL8qfrTs8Q+FkKvgrx7IdO6hdnWV4/PmmuLhXR9il2wZ/J/zh7ZHd3Aqb6YV3leaCBODch+f+6foDSbyRLOz6m5pagVvRjHJKbpveeRwwpWUQx70sAlrmKvyoSiKul2qhKLNKP4EkrmA3EQkBQ= AAACdHichVG7SgNBFD1Z3/GRqI2gRXCJaBMmIihWQRtLE00UfCy762gG98XuJBiDP+APWKRSFBE/w8YfsPATxDKijYU3mwVRUe8yO2fO3HPnzFzDs0QgGXuMKR2dXd09vX3x/oHBoURyeKQUuBXf5EXTtVx/09ADbgmHF6WQFt/0fK7bhsU3jMPl1v5GlfuBcJ11WfP4jq0fOGJfmLokSksmytrR9NFufdvzhc1PZrSkyjIsjNRPkI2AiihW3eQ1trEHFyYqsMHhQBK2oCOgbwtZMHjE7aBOnE9IhPscJ4iTtkJZnDJ0Yg/pf0CrrYh1aN2qGYRqk06xaPikTCHNHtgNa7J7dsue2PuvtephjZaXGs1GW8s9LXE6tvb2r8qmWaL8qfrTs8Q+FkKvgrx7IdO6hdnWV4/PmmuLhXR9il2wZ/J/zh7ZHd3Aqb6YV3leaCBODch+f+6foDSbyRLOz6m5pagVvRjHJKbpveeRwwpWUQx70sAlrmKvyoSiKul2qhKLNKP4EkrmA3EQkBQ= AAACdHichVG7SgNBFD1Z3/GRqI2gRXCJaBMmIihWQRtLE00UfCy762gG98XuJBiDP+APWKRSFBE/w8YfsPATxDKijYU3mwVRUe8yO2fO3HPnzFzDs0QgGXuMKR2dXd09vX3x/oHBoURyeKQUuBXf5EXTtVx/09ADbgmHF6WQFt/0fK7bhsU3jMPl1v5GlfuBcJ11WfP4jq0fOGJfmLokSksmytrR9NFufdvzhc1PZrSkyjIsjNRPkI2AiihW3eQ1trEHFyYqsMHhQBK2oCOgbwtZMHjE7aBOnE9IhPscJ4iTtkJZnDJ0Yg/pf0CrrYh1aN2qGYRqk06xaPikTCHNHtgNa7J7dsue2PuvtephjZaXGs1GW8s9LXE6tvb2r8qmWaL8qfrTs8Q+FkKvgrx7IdO6hdnWV4/PmmuLhXR9il2wZ/J/zh7ZHd3Aqb6YV3leaCBODch+f+6foDSbyRLOz6m5pagVvRjHJKbpveeRwwpWUQx70sAlrmKvyoSiKul2qhKLNKP4EkrmA3EQkBQ= ' AAACaXichVFNLwNBGH66vuujxaXhIjbEqZmKhDgJF8cWLYk2srumNbpfdqdNaPwBJzfBiURE/AwXf8ChP0EcSVwcvLvdRGjwTmbmmWfe551nZnTXFL5krBlTOjq7unt6++L9A4NDieTwSMF3ap7B84ZjOt6WrvncFDbPSyFNvuV6XLN0k2/q1ZVgf7POPV849oY8dHnJ0iq2KAtDk0QVir6w+MFOUmVpFsZEO8hEQEUUWSd5iyJ24cBADRY4bEjCJjT41LaRAYNLXAkN4jxCItznOEactDXK4pShEVulsUKr7Yi1aR3U9EO1QaeY1D1STmCKPbE79soe2T17Zh+/1mqENQIvhzTrLS13dxInqfX3f1UWzRJ7X6o/PUuUsRB6FeTdDZngFkZLXz86e11fXJtqTLNr9kL+r1iTPdAN7PqbcZPja5eI0wdkfj53OyjMpjOEc3Pq0nL0Fb0YxyRm6L3nsYRVZJGnc/dxinNcxF6UYSWljLVSlVikGcW3UNRPvZ+MLQ== AAACaXichVFNLwNBGH66vuujxaXhIjbEqZmKhDgJF8cWLYk2srumNbpfdqdNaPwBJzfBiURE/AwXf8ChP0EcSVwcvLvdRGjwTmbmmWfe551nZnTXFL5krBlTOjq7unt6++L9A4NDieTwSMF3ap7B84ZjOt6WrvncFDbPSyFNvuV6XLN0k2/q1ZVgf7POPV849oY8dHnJ0iq2KAtDk0QVir6w+MFOUmVpFsZEO8hEQEUUWSd5iyJ24cBADRY4bEjCJjT41LaRAYNLXAkN4jxCItznOEactDXK4pShEVulsUKr7Yi1aR3U9EO1QaeY1D1STmCKPbE79soe2T17Zh+/1mqENQIvhzTrLS13dxInqfX3f1UWzRJ7X6o/PUuUsRB6FeTdDZngFkZLXz86e11fXJtqTLNr9kL+r1iTPdAN7PqbcZPja5eI0wdkfj53OyjMpjOEc3Pq0nL0Fb0YxyRm6L3nsYRVZJGnc/dxinNcxF6UYSWljLVSlVikGcW3UNRPvZ+MLQ== AAACaXichVFNLwNBGH66vuujxaXhIjbEqZmKhDgJF8cWLYk2srumNbpfdqdNaPwBJzfBiURE/AwXf8ChP0EcSVwcvLvdRGjwTmbmmWfe551nZnTXFL5krBlTOjq7unt6++L9A4NDieTwSMF3ap7B84ZjOt6WrvncFDbPSyFNvuV6XLN0k2/q1ZVgf7POPV849oY8dHnJ0iq2KAtDk0QVir6w+MFOUmVpFsZEO8hEQEUUWSd5iyJ24cBADRY4bEjCJjT41LaRAYNLXAkN4jxCItznOEactDXK4pShEVulsUKr7Yi1aR3U9EO1QaeY1D1STmCKPbE79soe2T17Zh+/1mqENQIvhzTrLS13dxInqfX3f1UWzRJ7X6o/PUuUsRB6FeTdDZngFkZLXz86e11fXJtqTLNr9kL+r1iTPdAN7PqbcZPja5eI0wdkfj53OyjMpjOEc3Pq0nL0Fb0YxyRm6L3nsYRVZJGnc/dxinNcxF6UYSWljLVSlVikGcW3UNRPvZ+MLQ== AAACaXichVFNLwNBGH66vuujxaXhIjbEqZmKhDgJF8cWLYk2srumNbpfdqdNaPwBJzfBiURE/AwXf8ChP0EcSVwcvLvdRGjwTmbmmWfe551nZnTXFL5krBlTOjq7unt6++L9A4NDieTwSMF3ap7B84ZjOt6WrvncFDbPSyFNvuV6XLN0k2/q1ZVgf7POPV849oY8dHnJ0iq2KAtDk0QVir6w+MFOUmVpFsZEO8hEQEUUWSd5iyJ24cBADRY4bEjCJjT41LaRAYNLXAkN4jxCItznOEactDXK4pShEVulsUKr7Yi1aR3U9EO1QaeY1D1STmCKPbE79soe2T17Zh+/1mqENQIvhzTrLS13dxInqfX3f1UWzRJ7X6o/PUuUsRB6FeTdDZngFkZLXz86e11fXJtqTLNr9kL+r1iTPdAN7PqbcZPja5eI0wdkfj53OyjMpjOEc3Pq0nL0Fb0YxyRm6L3nsYRVZJGnc/dxinNcxF6UYSWljLVSlVikGcW3UNRPvZ+MLQ== [-1, 0, 0.2, 10] [1, 0, 1, 1] ⾮ゼロ → 1 = 0 + M X i=1 ix0 i AAAClnichVFdSxtBFD2u2mrUmrYvQl8Gg1IoyN1SaCm0SIvYl0JMjAkYs+yukzi4X+xugnbJH/AP+OCTgoj0N/hUaPsHfPAniI8W+tKH3mwWpAbrXXbmzLn33DkzYwWOimKiiyFteGT0wcOx8dzE5NSj6fzjJ2uR3w5tWbF9xw9rlhlJR3myEqvYkbUglKZrObJqbX/s5asdGUbK91bj3UBuuGbLU01lmzFTRl4X70Q92FLCIPFC1KO2K4xEMal3G5+zjBI7jaQehMqVXUMZ+QItUBpiEOgZKCCLop8/QR2b8GGjDRcSHmLGDkxE/K1DByFgbgMJcyEjleYlusixts1VkitMZrd5bPFqPWM9Xvd6Rqna5l0c/kNWCszROZ3SNf2kr3RJf+7slaQ9el52ebb6WhkY03sz5d/3qlyeY2zdqP7rOUYTb1Kvir0HKdM7hd3Xd77sX5ffluaSeTqiK/Z/SBf0jU/gdX7ZxyuydIAcP4B++7oHwdrLBZ3xyqvC4ofsKcbwDLN4zvf9Gov4hCIqvO8+zvAdP7QZ7b22pC33S7WhTPMU/4RW/Au3ypqE AAAClnichVFdSxtBFD2u2mrUmrYvQl8Gg1IoyN1SaCm0SIvYl0JMjAkYs+yukzi4X+xugnbJH/AP+OCTgoj0N/hUaPsHfPAniI8W+tKH3mwWpAbrXXbmzLn33DkzYwWOimKiiyFteGT0wcOx8dzE5NSj6fzjJ2uR3w5tWbF9xw9rlhlJR3myEqvYkbUglKZrObJqbX/s5asdGUbK91bj3UBuuGbLU01lmzFTRl4X70Q92FLCIPFC1KO2K4xEMal3G5+zjBI7jaQehMqVXUMZ+QItUBpiEOgZKCCLop8/QR2b8GGjDRcSHmLGDkxE/K1DByFgbgMJcyEjleYlusixts1VkitMZrd5bPFqPWM9Xvd6Rqna5l0c/kNWCszROZ3SNf2kr3RJf+7slaQ9el52ebb6WhkY03sz5d/3qlyeY2zdqP7rOUYTb1Kvir0HKdM7hd3Xd77sX5ffluaSeTqiK/Z/SBf0jU/gdX7ZxyuydIAcP4B++7oHwdrLBZ3xyqvC4ofsKcbwDLN4zvf9Gov4hCIqvO8+zvAdP7QZ7b22pC33S7WhTPMU/4RW/Au3ypqE AAAClnichVFdSxtBFD2u2mrUmrYvQl8Gg1IoyN1SaCm0SIvYl0JMjAkYs+yukzi4X+xugnbJH/AP+OCTgoj0N/hUaPsHfPAniI8W+tKH3mwWpAbrXXbmzLn33DkzYwWOimKiiyFteGT0wcOx8dzE5NSj6fzjJ2uR3w5tWbF9xw9rlhlJR3myEqvYkbUglKZrObJqbX/s5asdGUbK91bj3UBuuGbLU01lmzFTRl4X70Q92FLCIPFC1KO2K4xEMal3G5+zjBI7jaQehMqVXUMZ+QItUBpiEOgZKCCLop8/QR2b8GGjDRcSHmLGDkxE/K1DByFgbgMJcyEjleYlusixts1VkitMZrd5bPFqPWM9Xvd6Rqna5l0c/kNWCszROZ3SNf2kr3RJf+7slaQ9el52ebb6WhkY03sz5d/3qlyeY2zdqP7rOUYTb1Kvir0HKdM7hd3Xd77sX5ffluaSeTqiK/Z/SBf0jU/gdX7ZxyuydIAcP4B++7oHwdrLBZ3xyqvC4ofsKcbwDLN4zvf9Gov4hCIqvO8+zvAdP7QZ7b22pC33S7WhTPMU/4RW/Au3ypqE AAAClnichVFdSxtBFD2u2mrUmrYvQl8Gg1IoyN1SaCm0SIvYl0JMjAkYs+yukzi4X+xugnbJH/AP+OCTgoj0N/hUaPsHfPAniI8W+tKH3mwWpAbrXXbmzLn33DkzYwWOimKiiyFteGT0wcOx8dzE5NSj6fzjJ2uR3w5tWbF9xw9rlhlJR3myEqvYkbUglKZrObJqbX/s5asdGUbK91bj3UBuuGbLU01lmzFTRl4X70Q92FLCIPFC1KO2K4xEMal3G5+zjBI7jaQehMqVXUMZ+QItUBpiEOgZKCCLop8/QR2b8GGjDRcSHmLGDkxE/K1DByFgbgMJcyEjleYlusixts1VkitMZrd5bPFqPWM9Xvd6Rqna5l0c/kNWCszROZ3SNf2kr3RJf+7slaQ9el52ebb6WhkY03sz5d/3qlyeY2zdqP7rOUYTb1Kvir0HKdM7hd3Xd77sX5ffluaSeTqiK/Z/SBf0jU/gdX7ZxyuydIAcP4B++7oHwdrLBZ3xyqvC4ofsKcbwDLN4zvf9Gov4hCIqvO8+zvAdP7QZ7b22pC33S7WhTPMU/4RW/Au3ypqE 1. xの近くで簡単化 2. 加法モデルの係数 （貢献度）を決める ブラックボックス モデルの世界

LIME, SHAP 25 貢献度は満たすべき条件から考える [Lundberg, Lee, 17] 1. 局所的正確性 : local accuracy p データ点の直上では出⼒が⼀致 2. 値がゼロの変数の無影響性 : missingness p データ点で値がゼロの変数は貢献度ゼロ 3. 複数モデル⽐較での無⽭盾性 : consistency p 簡単化の前後でモデル間の貢献度の⼤⼩関係が同じ f(x) = g(x0) AAACenichVHLSgMxFD0dX7U+WhVBcDNYlBahpFJQBKHgxqVVWwUfZWZMa3BezEyLdegP+AMuXCmIaD/DjT/gwk8QlwpuuvB2OiAq6g1JTk7uuTlJVFsXrsfYU0Tq6e3rH4gOxoaGR0bjibHxkmvVHI0XNUu3nB1VcbkuTF70hKfzHdvhiqHqfFs9Xu3sb9e54wrL3PIaNt83lKopKkJTPKLKiclK6iQtr8jV1MmBv2c7wuDNtFxOJFmGBSH/BNkQJBHGupW4wR4OYUFDDQY4THiEdShwqe0iCwabuH34xDmERLDP0USMtDXK4pShEHtMY5VWuyFr0rpT0w3UGp2iU3dIKWOWPbJb9soeWIs9s/avtfygRsdLg2a1q+V2OX42tfn+r8qg2cPRp+pPzx4qWAq8CvJuB0znFlpXXz89f91c3pj159gVeyH/l+yJ3dMNzPqbdl3gGxeI0Qdkvz/3T1BayGQJF3LJfC78iiimMYMUvfci8ljDOop0ro9L3KEVaUszUlqa76ZKkVAzgS8h5T4ATeORNg== AAACenichVHLSgMxFD0dX7U+WhVBcDNYlBahpFJQBKHgxqVVWwUfZWZMa3BezEyLdegP+AMuXCmIaD/DjT/gwk8QlwpuuvB2OiAq6g1JTk7uuTlJVFsXrsfYU0Tq6e3rH4gOxoaGR0bjibHxkmvVHI0XNUu3nB1VcbkuTF70hKfzHdvhiqHqfFs9Xu3sb9e54wrL3PIaNt83lKopKkJTPKLKiclK6iQtr8jV1MmBv2c7wuDNtFxOJFmGBSH/BNkQJBHGupW4wR4OYUFDDQY4THiEdShwqe0iCwabuH34xDmERLDP0USMtDXK4pShEHtMY5VWuyFr0rpT0w3UGp2iU3dIKWOWPbJb9soeWIs9s/avtfygRsdLg2a1q+V2OX42tfn+r8qg2cPRp+pPzx4qWAq8CvJuB0znFlpXXz89f91c3pj159gVeyH/l+yJ3dMNzPqbdl3gGxeI0Qdkvz/3T1BayGQJF3LJfC78iiimMYMUvfci8ljDOop0ro9L3KEVaUszUlqa76ZKkVAzgS8h5T4ATeORNg== AAACenichVHLSgMxFD0dX7U+WhVBcDNYlBahpFJQBKHgxqVVWwUfZWZMa3BezEyLdegP+AMuXCmIaD/DjT/gwk8QlwpuuvB2OiAq6g1JTk7uuTlJVFsXrsfYU0Tq6e3rH4gOxoaGR0bjibHxkmvVHI0XNUu3nB1VcbkuTF70hKfzHdvhiqHqfFs9Xu3sb9e54wrL3PIaNt83lKopKkJTPKLKiclK6iQtr8jV1MmBv2c7wuDNtFxOJFmGBSH/BNkQJBHGupW4wR4OYUFDDQY4THiEdShwqe0iCwabuH34xDmERLDP0USMtDXK4pShEHtMY5VWuyFr0rpT0w3UGp2iU3dIKWOWPbJb9soeWIs9s/avtfygRsdLg2a1q+V2OX42tfn+r8qg2cPRp+pPzx4qWAq8CvJuB0znFlpXXz89f91c3pj159gVeyH/l+yJ3dMNzPqbdl3gGxeI0Qdkvz/3T1BayGQJF3LJfC78iiimMYMUvfci8ljDOop0ro9L3KEVaUszUlqa76ZKkVAzgS8h5T4ATeORNg== AAACenichVHLSgMxFD0dX7U+WhVBcDNYlBahpFJQBKHgxqVVWwUfZWZMa3BezEyLdegP+AMuXCmIaD/DjT/gwk8QlwpuuvB2OiAq6g1JTk7uuTlJVFsXrsfYU0Tq6e3rH4gOxoaGR0bjibHxkmvVHI0XNUu3nB1VcbkuTF70hKfzHdvhiqHqfFs9Xu3sb9e54wrL3PIaNt83lKopKkJTPKLKiclK6iQtr8jV1MmBv2c7wuDNtFxOJFmGBSH/BNkQJBHGupW4wR4OYUFDDQY4THiEdShwqe0iCwabuH34xDmERLDP0USMtDXK4pShEHtMY5VWuyFr0rpT0w3UGp2iU3dIKWOWPbJb9soeWIs9s/avtfygRsdLg2a1q+V2OX42tfn+r8qg2cPRp+pPzx4qWAq8CvJuB0znFlpXXz89f91c3pj159gVeyH/l+yJ3dMNzPqbdl3gGxeI0Qdkvz/3T1BayGQJF3LJfC78iiimMYMUvfci8ljDOop0ro9L3KEVaUszUlqa76ZKkVAzgS8h5T4ATeORNg== xi = 0 ) i = 0 AAAChHichVHLLgRBFD3ae7wGG4mFjgmxELnjESIhEhtLhkFiZNLdykxFT3enu2Y8JpY2fsDCikREbPkCGz9g4RPEksTGwp2eTgTBrVTVqVP33DpVZXq2DBTRY41WW1ff0NjUHGtpbWvviHd2rQZu0bdE2nJt1183jUDY0hFpJZUt1j1fGAXTFmvmznxlf60k/EC6zora98Rmwcg5cltahmIqG+/by0p9Ric9k5K5vDJ8393VM15e6lU+G0/QCIWh/wTJCCQQxaIbv0QGW3BhoYgCBBwoxjYMBNw2kATBY24TZeZ8RjLcFzhEjLVFzhKcYTC7w2OOVxsR6/C6UjMI1RafYnP3WaljgB7oil7onq7pid5/rVUOa1S87PNsVrXCy3Yc9yy//asq8KyQ/1T96VlhG1OhV8nevZCp3MKq6ksHJy/L06mB8iCd0zP7P6NHuuMbOKVX62JJpE4R4w9Ifn/un2B1dCTJeGk8MTcefUUTetGPIX7vScxhAYtI87lHuMYNbrUGbVgb0yaqqVpNpOnGl9BmPwAqV5O0 AAAChHichVHLLgRBFD3ae7wGG4mFjgmxELnjESIhEhtLhkFiZNLdykxFT3enu2Y8JpY2fsDCikREbPkCGz9g4RPEksTGwp2eTgTBrVTVqVP33DpVZXq2DBTRY41WW1ff0NjUHGtpbWvviHd2rQZu0bdE2nJt1183jUDY0hFpJZUt1j1fGAXTFmvmznxlf60k/EC6zora98Rmwcg5cltahmIqG+/by0p9Ric9k5K5vDJ8393VM15e6lU+G0/QCIWh/wTJCCQQxaIbv0QGW3BhoYgCBBwoxjYMBNw2kATBY24TZeZ8RjLcFzhEjLVFzhKcYTC7w2OOVxsR6/C6UjMI1RafYnP3WaljgB7oil7onq7pid5/rVUOa1S87PNsVrXCy3Yc9yy//asq8KyQ/1T96VlhG1OhV8nevZCp3MKq6ksHJy/L06mB8iCd0zP7P6NHuuMbOKVX62JJpE4R4w9Ifn/un2B1dCTJeGk8MTcefUUTetGPIX7vScxhAYtI87lHuMYNbrUGbVgb0yaqqVpNpOnGl9BmPwAqV5O0 AAAChHichVHLLgRBFD3ae7wGG4mFjgmxELnjESIhEhtLhkFiZNLdykxFT3enu2Y8JpY2fsDCikREbPkCGz9g4RPEksTGwp2eTgTBrVTVqVP33DpVZXq2DBTRY41WW1ff0NjUHGtpbWvviHd2rQZu0bdE2nJt1183jUDY0hFpJZUt1j1fGAXTFmvmznxlf60k/EC6zora98Rmwcg5cltahmIqG+/by0p9Ric9k5K5vDJ8393VM15e6lU+G0/QCIWh/wTJCCQQxaIbv0QGW3BhoYgCBBwoxjYMBNw2kATBY24TZeZ8RjLcFzhEjLVFzhKcYTC7w2OOVxsR6/C6UjMI1RafYnP3WaljgB7oil7onq7pid5/rVUOa1S87PNsVrXCy3Yc9yy//asq8KyQ/1T96VlhG1OhV8nevZCp3MKq6ksHJy/L06mB8iCd0zP7P6NHuuMbOKVX62JJpE4R4w9Ifn/un2B1dCTJeGk8MTcefUUTetGPIX7vScxhAYtI87lHuMYNbrUGbVgb0yaqqVpNpOnGl9BmPwAqV5O0 AAAChHichVHLLgRBFD3ae7wGG4mFjgmxELnjESIhEhtLhkFiZNLdykxFT3enu2Y8JpY2fsDCikREbPkCGz9g4RPEksTGwp2eTgTBrVTVqVP33DpVZXq2DBTRY41WW1ff0NjUHGtpbWvviHd2rQZu0bdE2nJt1183jUDY0hFpJZUt1j1fGAXTFmvmznxlf60k/EC6zora98Rmwcg5cltahmIqG+/by0p9Ric9k5K5vDJ8393VM15e6lU+G0/QCIWh/wTJCCQQxaIbv0QGW3BhoYgCBBwoxjYMBNw2kATBY24TZeZ8RjLcFzhEjLVFzhKcYTC7w2OOVxsR6/C6UjMI1RafYnP3WaljgB7oil7onq7pid5/rVUOa1S87PNsVrXCy3Yc9yy//asq8KyQ/1T96VlhG1OhV8nevZCp3MKq6ksHJy/L06mB8iCd0zP7P6NHuuMbOKVX62JJpE4R4w9Ifn/un2B1dCTJeGk8MTcefUUTetGPIX7vScxhAYtI87lHuMYNbrUGbVgb0yaqqVpNpOnGl9BmPwAqV5O0 fA(x) fA(x \ xi) fB(x) fB(x \ xi) ) i(fA, x) i(fB, x) AAAC1HichVFNLwNBGH67vuuruEhcJhpSCc1UJMSJujhSWhIrm901bSf2y+60SjmJi4OrgxOJiPgZLv6Ag58gjiQOHLy7XYQG72Znnnne53nnnRnNMbgnKL2PSA2NTc0trW3R9o7Oru5YT2/Os0uuzrK6bdjuqqZ6zOAWywouDLbquEw1NYOtaJtzfn6lzFyP29ay2HHYuqkWLJ7nuiqQUmJ2XplNVEbIGAkAkT0mTG6VPFJR+AiRC2wLM+kPSbpOEpUzvFAUquva20R2ilzhCSw1Siqh+5NL+5wSi9MkDYLUg1QI4hDGgh27BBk2wAYdSmACAwsEYgNU8PBbgxRQcJBbhypyLiIe5BnsQxS9JVQxVKjIbuJYwNVayFq49mt6gVvHXQz8XXQSGKJ39Io+0Vt6TR/o26+1qkENv5cdnLWalzlK91H/0su/LhNnAcUv1589C8jDVNArx96dgPFPodf85d2Tp6XpzFB1mJ7TR+z/jN7TGzyBVX7WLxZZ5hSi+ACpn9ddD3LjyRTixYn4zET4FK0wAIOQwPuehBmYhwXI4r538BppjDRJOWlPOpAOa1IpEnr64FtIx+9Udq8U AAAC1HichVFNLwNBGH67vuuruEhcJhpSCc1UJMSJujhSWhIrm901bSf2y+60SjmJi4OrgxOJiPgZLv6Ag58gjiQOHLy7XYQG72Znnnne53nnnRnNMbgnKL2PSA2NTc0trW3R9o7Oru5YT2/Os0uuzrK6bdjuqqZ6zOAWywouDLbquEw1NYOtaJtzfn6lzFyP29ay2HHYuqkWLJ7nuiqQUmJ2XplNVEbIGAkAkT0mTG6VPFJR+AiRC2wLM+kPSbpOEpUzvFAUquva20R2ilzhCSw1Siqh+5NL+5wSi9MkDYLUg1QI4hDGgh27BBk2wAYdSmACAwsEYgNU8PBbgxRQcJBbhypyLiIe5BnsQxS9JVQxVKjIbuJYwNVayFq49mt6gVvHXQz8XXQSGKJ39Io+0Vt6TR/o26+1qkENv5cdnLWalzlK91H/0su/LhNnAcUv1589C8jDVNArx96dgPFPodf85d2Tp6XpzFB1mJ7TR+z/jN7TGzyBVX7WLxZZ5hSi+ACpn9ddD3LjyRTixYn4zET4FK0wAIOQwPuehBmYhwXI4r538BppjDRJOWlPOpAOa1IpEnr64FtIx+9Udq8U AAAC1HichVFNLwNBGH67vuuruEhcJhpSCc1UJMSJujhSWhIrm901bSf2y+60SjmJi4OrgxOJiPgZLv6Ag58gjiQOHLy7XYQG72Znnnne53nnnRnNMbgnKL2PSA2NTc0trW3R9o7Oru5YT2/Os0uuzrK6bdjuqqZ6zOAWywouDLbquEw1NYOtaJtzfn6lzFyP29ay2HHYuqkWLJ7nuiqQUmJ2XplNVEbIGAkAkT0mTG6VPFJR+AiRC2wLM+kPSbpOEpUzvFAUquva20R2ilzhCSw1Siqh+5NL+5wSi9MkDYLUg1QI4hDGgh27BBk2wAYdSmACAwsEYgNU8PBbgxRQcJBbhypyLiIe5BnsQxS9JVQxVKjIbuJYwNVayFq49mt6gVvHXQz8XXQSGKJ39Io+0Vt6TR/o26+1qkENv5cdnLWalzlK91H/0su/LhNnAcUv1589C8jDVNArx96dgPFPodf85d2Tp6XpzFB1mJ7TR+z/jN7TGzyBVX7WLxZZ5hSi+ACpn9ddD3LjyRTixYn4zET4FK0wAIOQwPuehBmYhwXI4r538BppjDRJOWlPOpAOa1IpEnr64FtIx+9Udq8U AAAC1HichVFNLwNBGH67vuuruEhcJhpSCc1UJMSJujhSWhIrm901bSf2y+60SjmJi4OrgxOJiPgZLv6Ag58gjiQOHLy7XYQG72Znnnne53nnnRnNMbgnKL2PSA2NTc0trW3R9o7Oru5YT2/Os0uuzrK6bdjuqqZ6zOAWywouDLbquEw1NYOtaJtzfn6lzFyP29ay2HHYuqkWLJ7nuiqQUmJ2XplNVEbIGAkAkT0mTG6VPFJR+AiRC2wLM+kPSbpOEpUzvFAUquva20R2ilzhCSw1Siqh+5NL+5wSi9MkDYLUg1QI4hDGgh27BBk2wAYdSmACAwsEYgNU8PBbgxRQcJBbhypyLiIe5BnsQxS9JVQxVKjIbuJYwNVayFq49mt6gVvHXQz8XXQSGKJ39Io+0Vt6TR/o26+1qkENv5cdnLWalzlK91H/0su/LhNnAcUv1589C8jDVNArx96dgPFPodf85d2Tp6XpzFB1mJ7TR+z/jN7TGzyBVX7WLxZZ5hSi+ACpn9ddD3LjyRTixYn4zET4FK0wAIOQwPuehBmYhwXI4r538BppjDRJOWlPOpAOa1IpEnr64FtIx+9Udq8U

LIME, SHAP 26 [Lundberg, Lee, 17] p SHAP（SHapley Additive exPlanation）value ü 条件をすべて満たす貢献度の組 ü 協⼒ゲーム理論ではShapley valueとして知られる p LIME ü 貢献度は最適化問題を解くことで算出 i(f, x) = X z0✓x0 |z0|!(M |z0| 1)! M! [fx(z0) fx(z0 \ i)] AAAC/HichVHNahRBEK4Zo4nrT1a9CLn0ukR2IRt6QkARhIAXL4H8uElgdx1mxpqkyfxlumfZZLI+gC+QgydDQgghV1/Aiy/gIY8gHiN4UbBmdnCNQa2mu7/+qr6q6m478oRUnJ9p+pWRq9dGx66Xbty8dXu8fOfuigyT2MGmE3phvGZbEj0RYFMJ5eFaFKPl2x6u2pvPMv9qF2MpwuCF2o6w41vrgXCFYymizPJeO9oQzBSs5k6xXp09ZW2Z+MxMd16m7SgWPvYzxpaocIv1fpHEurHlpLvDuN1KbZ412O8Maxj1Sj+dr/RLLdfs1YauOkVeZKgMKl8EiWSi3jHLVT7Nc2OXgVGAKhS2EJaPoA2vIAQHEvABIQBF2AMLJI0WGMAhIq4DKXExIZH7EfpQIm1CUUgRFrGbtK7TqVWwAZ2znDJXO1TFoxmTksEk/8SP+Tn/yE/4Z/79r7nSPEfWyzbt9kCLkTn+5v7yt/+qfNoVbAxV/+xZgQuP814F9R7lTHYLZ6Dv7uydLz9Zmkwf8n3+hfp/x8/4B7pB0P3qHC7i0lso0QcYfz73ZbAyM20QXpytzs0WXzEGE/AAavTej2AOnsMCNKnuD62qTWkN/bV+oJ/op4NQXSs09+CC6e9/AgbZwf0= AAAC/HichVHNahRBEK4Zo4nrT1a9CLn0ukR2IRt6QkARhIAXL4H8uElgdx1mxpqkyfxlumfZZLI+gC+QgydDQgghV1/Aiy/gIY8gHiN4UbBmdnCNQa2mu7/+qr6q6m478oRUnJ9p+pWRq9dGx66Xbty8dXu8fOfuigyT2MGmE3phvGZbEj0RYFMJ5eFaFKPl2x6u2pvPMv9qF2MpwuCF2o6w41vrgXCFYymizPJeO9oQzBSs5k6xXp09ZW2Z+MxMd16m7SgWPvYzxpaocIv1fpHEurHlpLvDuN1KbZ412O8Maxj1Sj+dr/RLLdfs1YauOkVeZKgMKl8EiWSi3jHLVT7Nc2OXgVGAKhS2EJaPoA2vIAQHEvABIQBF2AMLJI0WGMAhIq4DKXExIZH7EfpQIm1CUUgRFrGbtK7TqVWwAZ2znDJXO1TFoxmTksEk/8SP+Tn/yE/4Z/79r7nSPEfWyzbt9kCLkTn+5v7yt/+qfNoVbAxV/+xZgQuP814F9R7lTHYLZ6Dv7uydLz9Zmkwf8n3+hfp/x8/4B7pB0P3qHC7i0lso0QcYfz73ZbAyM20QXpytzs0WXzEGE/AAavTej2AOnsMCNKnuD62qTWkN/bV+oJ/op4NQXSs09+CC6e9/AgbZwf0= AAAC/HichVHNahRBEK4Zo4nrT1a9CLn0ukR2IRt6QkARhIAXL4H8uElgdx1mxpqkyfxlumfZZLI+gC+QgydDQgghV1/Aiy/gIY8gHiN4UbBmdnCNQa2mu7/+qr6q6m478oRUnJ9p+pWRq9dGx66Xbty8dXu8fOfuigyT2MGmE3phvGZbEj0RYFMJ5eFaFKPl2x6u2pvPMv9qF2MpwuCF2o6w41vrgXCFYymizPJeO9oQzBSs5k6xXp09ZW2Z+MxMd16m7SgWPvYzxpaocIv1fpHEurHlpLvDuN1KbZ412O8Maxj1Sj+dr/RLLdfs1YauOkVeZKgMKl8EiWSi3jHLVT7Nc2OXgVGAKhS2EJaPoA2vIAQHEvABIQBF2AMLJI0WGMAhIq4DKXExIZH7EfpQIm1CUUgRFrGbtK7TqVWwAZ2znDJXO1TFoxmTksEk/8SP+Tn/yE/4Z/79r7nSPEfWyzbt9kCLkTn+5v7yt/+qfNoVbAxV/+xZgQuP814F9R7lTHYLZ6Dv7uydLz9Zmkwf8n3+hfp/x8/4B7pB0P3qHC7i0lso0QcYfz73ZbAyM20QXpytzs0WXzEGE/AAavTej2AOnsMCNKnuD62qTWkN/bV+oJ/op4NQXSs09+CC6e9/AgbZwf0= AAAC/HichVHNahRBEK4Zo4nrT1a9CLn0ukR2IRt6QkARhIAXL4H8uElgdx1mxpqkyfxlumfZZLI+gC+QgydDQgghV1/Aiy/gIY8gHiN4UbBmdnCNQa2mu7/+qr6q6m478oRUnJ9p+pWRq9dGx66Xbty8dXu8fOfuigyT2MGmE3phvGZbEj0RYFMJ5eFaFKPl2x6u2pvPMv9qF2MpwuCF2o6w41vrgXCFYymizPJeO9oQzBSs5k6xXp09ZW2Z+MxMd16m7SgWPvYzxpaocIv1fpHEurHlpLvDuN1KbZ412O8Maxj1Sj+dr/RLLdfs1YauOkVeZKgMKl8EiWSi3jHLVT7Nc2OXgVGAKhS2EJaPoA2vIAQHEvABIQBF2AMLJI0WGMAhIq4DKXExIZH7EfpQIm1CUUgRFrGbtK7TqVWwAZ2znDJXO1TFoxmTksEk/8SP+Tn/yE/4Z/79r7nSPEfWyzbt9kCLkTn+5v7yt/+qfNoVbAxV/+xZgQuP814F9R7lTHYLZ6Dv7uydLz9Zmkwf8n3+hfp/x8/4B7pB0P3qHC7i0lso0QcYfz73ZbAyM20QXpytzs0WXzEGE/AAavTej2AOnsMCNKnuD62qTWkN/bV+oJ/op4NQXSs09+CC6e9/AgbZwf0= fx(z0) = f(hx(z0)) = E[f(z)|zS] AAACl3ichVFNLwNBGH6s7/oqLsSl0ZC6yLRpQiRCIsRRVZFUbXbXLBP7ld1tQ8sf8AdInEhExH9wceAPOPgJ4kji4uDd7SaC4J3MzDPPvM87z8yojiE8n7HHBqmxqbmlta091tHZ1d0T7+1b9eyyq/GCZhu2u64qHjeExQu+8A2+7rhcMVWDr6m7c8H+WoW7nrCtFX/f4SVT2baELjTFJ0qOZ3R5L1XdrG04rjD54VhiOqGndr5yATlf1FPVsYOqnC/J8SQbZ2EkfoJ0BJKIYsmOX2IDW7ChoQwTHBZ8wgYUeNSKSIPBIa6EGnEuIRHucxwiRtoyZXHKUIjdpXGbVsWItWgd1PRCtUanGNRdUiYwwh7YFXth9+yaPbH3X2vVwhqBl32a1bqWO3LP0UD+7V+VSbOPnU/Vn5596JgMvQry7oRMcAutrq9Uj1/yU8sjtVF2zp7J/xl7ZLd0A6vyql3k+PIpYvQB6e/P/ROsZsbThHPZ5Gw2+oo2DGEYKXrvCcxiEUso0LknuMEd7qVBaUZakBbrqVJDpOnHl5ByH7l7m8U= AAACl3ichVFNLwNBGH6s7/oqLsSl0ZC6yLRpQiRCIsRRVZFUbXbXLBP7ld1tQ8sf8AdInEhExH9wceAPOPgJ4kji4uDd7SaC4J3MzDPPvM87z8yojiE8n7HHBqmxqbmlta091tHZ1d0T7+1b9eyyq/GCZhu2u64qHjeExQu+8A2+7rhcMVWDr6m7c8H+WoW7nrCtFX/f4SVT2baELjTFJ0qOZ3R5L1XdrG04rjD54VhiOqGndr5yATlf1FPVsYOqnC/J8SQbZ2EkfoJ0BJKIYsmOX2IDW7ChoQwTHBZ8wgYUeNSKSIPBIa6EGnEuIRHucxwiRtoyZXHKUIjdpXGbVsWItWgd1PRCtUanGNRdUiYwwh7YFXth9+yaPbH3X2vVwhqBl32a1bqWO3LP0UD+7V+VSbOPnU/Vn5596JgMvQry7oRMcAutrq9Uj1/yU8sjtVF2zp7J/xl7ZLd0A6vyql3k+PIpYvQB6e/P/ROsZsbThHPZ5Gw2+oo2DGEYKXrvCcxiEUso0LknuMEd7qVBaUZakBbrqVJDpOnHl5ByH7l7m8U= AAACl3ichVFNLwNBGH6s7/oqLsSl0ZC6yLRpQiRCIsRRVZFUbXbXLBP7ld1tQ8sf8AdInEhExH9wceAPOPgJ4kji4uDd7SaC4J3MzDPPvM87z8yojiE8n7HHBqmxqbmlta091tHZ1d0T7+1b9eyyq/GCZhu2u64qHjeExQu+8A2+7rhcMVWDr6m7c8H+WoW7nrCtFX/f4SVT2baELjTFJ0qOZ3R5L1XdrG04rjD54VhiOqGndr5yATlf1FPVsYOqnC/J8SQbZ2EkfoJ0BJKIYsmOX2IDW7ChoQwTHBZ8wgYUeNSKSIPBIa6EGnEuIRHucxwiRtoyZXHKUIjdpXGbVsWItWgd1PRCtUanGNRdUiYwwh7YFXth9+yaPbH3X2vVwhqBl32a1bqWO3LP0UD+7V+VSbOPnU/Vn5596JgMvQry7oRMcAutrq9Uj1/yU8sjtVF2zp7J/xl7ZLd0A6vyql3k+PIpYvQB6e/P/ROsZsbThHPZ5Gw2+oo2DGEYKXrvCcxiEUso0LknuMEd7qVBaUZakBbrqVJDpOnHl5ByH7l7m8U= AAACl3ichVFNLwNBGH6s7/oqLsSl0ZC6yLRpQiRCIsRRVZFUbXbXLBP7ld1tQ8sf8AdInEhExH9wceAPOPgJ4kji4uDd7SaC4J3MzDPPvM87z8yojiE8n7HHBqmxqbmlta091tHZ1d0T7+1b9eyyq/GCZhu2u64qHjeExQu+8A2+7rhcMVWDr6m7c8H+WoW7nrCtFX/f4SVT2baELjTFJ0qOZ3R5L1XdrG04rjD54VhiOqGndr5yATlf1FPVsYOqnC/J8SQbZ2EkfoJ0BJKIYsmOX2IDW7ChoQwTHBZ8wgYUeNSKSIPBIa6EGnEuIRHucxwiRtoyZXHKUIjdpXGbVsWItWgd1PRCtUanGNRdUiYwwh7YFXth9+yaPbH3X2vVwhqBl32a1bqWO3LP0UD+7V+VSbOPnU/Vn5596JgMvQry7oRMcAutrq9Uj1/yU8sjtVF2zp7J/xl7ZLd0A6vyql3k+PIpYvQB6e/P/ROsZsbThHPZ5Gw2+oo2DGEYKXrvCcxiEUso0LknuMEd7qVBaUZakBbrqVJDpOnHl5ByH7l7m8U= 貢献度は満たすべき条件から考える ⇠ = argming2G L(f, g, ⇡x0 ) + ⌦(g) AAACtXichVFNSxwxGH4c+6Grrau9CL0EF2VFkWwRFEEQemgPha7aVcFZl8yYHYPzxUx2WTvMH+gf6MGThVKKd/9AL/0DPQheFYpHBS899N3ZgdKK9Q1Jnjx5nzdPEit0Vaw5P+0z+h88fPR4YLAwNPzk6UhxdGwjDlqRLWt24AbRliVi6Spf1rTSrtwKIyk8y5Wb1v7L7v5mW0axCvx3+iCUdU84vmoqW2iiGsWq2VFsmZladnQiIsdTfsoaicNM5TPTE3rPFm7yKk3Zm3JzljmzzAwVJXR2EjOMlCfTdJrNMPOtJx3Bys50o1jiczwLdhtUclBCHtWg+AUmdhHARgseJHxowi4EYmrbqIAjJK6OhLiIkMr2JVIUSNuiLEkZgth9Gh1abeesT+tuzThT23SKSz0iJcMk/8G/8iv+nR/zn/zXnbWSrEbXywHNVk8rw8bIh/H1m3tVHs0ae39U//Ws0cRi5lWR9zBjurewe/r2+49X60trk8kU/8Qvyf8RP+Xf6AZ++9r+vCrXDlGgD6j8+9y3wcaLuQrh1fnSynz+FQN4jgmU6b0XsILXqKJG557gDOe4MBaMurFrNHupRl+ueYa/wgh+A+I/p24= AAACtXichVFNSxwxGH4c+6Grrau9CL0EF2VFkWwRFEEQemgPha7aVcFZl8yYHYPzxUx2WTvMH+gf6MGThVKKd/9AL/0DPQheFYpHBS899N3ZgdKK9Q1Jnjx5nzdPEit0Vaw5P+0z+h88fPR4YLAwNPzk6UhxdGwjDlqRLWt24AbRliVi6Spf1rTSrtwKIyk8y5Wb1v7L7v5mW0axCvx3+iCUdU84vmoqW2iiGsWq2VFsmZladnQiIsdTfsoaicNM5TPTE3rPFm7yKk3Zm3JzljmzzAwVJXR2EjOMlCfTdJrNMPOtJx3Bys50o1jiczwLdhtUclBCHtWg+AUmdhHARgseJHxowi4EYmrbqIAjJK6OhLiIkMr2JVIUSNuiLEkZgth9Gh1abeesT+tuzThT23SKSz0iJcMk/8G/8iv+nR/zn/zXnbWSrEbXywHNVk8rw8bIh/H1m3tVHs0ae39U//Ws0cRi5lWR9zBjurewe/r2+49X60trk8kU/8Qvyf8RP+Xf6AZ++9r+vCrXDlGgD6j8+9y3wcaLuQrh1fnSynz+FQN4jgmU6b0XsILXqKJG557gDOe4MBaMurFrNHupRl+ueYa/wgh+A+I/p24= AAACtXichVFNSxwxGH4c+6Grrau9CL0EF2VFkWwRFEEQemgPha7aVcFZl8yYHYPzxUx2WTvMH+gf6MGThVKKd/9AL/0DPQheFYpHBS899N3ZgdKK9Q1Jnjx5nzdPEit0Vaw5P+0z+h88fPR4YLAwNPzk6UhxdGwjDlqRLWt24AbRliVi6Spf1rTSrtwKIyk8y5Wb1v7L7v5mW0axCvx3+iCUdU84vmoqW2iiGsWq2VFsmZladnQiIsdTfsoaicNM5TPTE3rPFm7yKk3Zm3JzljmzzAwVJXR2EjOMlCfTdJrNMPOtJx3Bys50o1jiczwLdhtUclBCHtWg+AUmdhHARgseJHxowi4EYmrbqIAjJK6OhLiIkMr2JVIUSNuiLEkZgth9Gh1abeesT+tuzThT23SKSz0iJcMk/8G/8iv+nR/zn/zXnbWSrEbXywHNVk8rw8bIh/H1m3tVHs0ae39U//Ws0cRi5lWR9zBjurewe/r2+49X60trk8kU/8Qvyf8RP+Xf6AZ++9r+vCrXDlGgD6j8+9y3wcaLuQrh1fnSynz+FQN4jgmU6b0XsILXqKJG557gDOe4MBaMurFrNHupRl+ueYa/wgh+A+I/p24= AAACtXichVFNSxwxGH4c+6Grrau9CL0EF2VFkWwRFEEQemgPha7aVcFZl8yYHYPzxUx2WTvMH+gf6MGThVKKd/9AL/0DPQheFYpHBS899N3ZgdKK9Q1Jnjx5nzdPEit0Vaw5P+0z+h88fPR4YLAwNPzk6UhxdGwjDlqRLWt24AbRliVi6Spf1rTSrtwKIyk8y5Wb1v7L7v5mW0axCvx3+iCUdU84vmoqW2iiGsWq2VFsmZladnQiIsdTfsoaicNM5TPTE3rPFm7yKk3Zm3JzljmzzAwVJXR2EjOMlCfTdJrNMPOtJx3Bys50o1jiczwLdhtUclBCHtWg+AUmdhHARgseJHxowi4EYmrbqIAjJK6OhLiIkMr2JVIUSNuiLEkZgth9Gh1abeesT+tuzThT23SKSz0iJcMk/8G/8iv+nR/zn/zXnbWSrEbXywHNVk8rw8bIh/H1m3tVHs0ae39U//Ws0cRi5lWR9zBjurewe/r2+49X60trk8kU/8Qvyf8RP+Xf6AZ++9r+vCrXDlGgD6j8+9y3wcaLuQrh1fnSynz+FQN4jgmU6b0XsILXqKJG557gDOe4MBaMurFrNHupRl+ueYa/wgh+A+I/p24=

SHAP : Python実装 27 • SHAP value • 著者によるPython実装がある ü Shapley valueの計算ならRでもいくつかライブラリがある • ブラックボックスモデルを作成 ü データセット : ボストンの家賃価格 ü モデル : RandomForest https://github.com/slundberg/shap

SHAP : Python実装 28 • SHAP value • Explaner : SHAP valueを使って予測を⾏う ü モデルの種類（treeベースなど）に特化したmethodが実装 予測値 バーの⻑さ : 特徴量の貢献分 ⾚ : 正の寄与, ⻘ : 負の寄与 https://github.com/slundberg/shap

機械学習モデルの結果を解釈する⽅法 29 1. どの特徴量が重要か p Feature Importance 2. 各特徴量が予測にどう影響するか p Partial Dependence p Surrogate Model : Appendix 3. ある予測結果に対して特徴量がどう寄与するか p LIME, SHAP p Grad-CAM, Grad-CAM++ ⼤きくは、以下の3つの説明ができること （モデルの説明 : 1, 2 結果の説明 : 3）

Grad-CAM, Grad-CAM++ 30 DNNに対して予測への影響が⼤きい箇所を可視化 • Class Activation Mapping (CAM) : 特定のクラスに反応する領域を可視化 • Grad-CAM : 勾配ベース(Back-propagation)のCAM ⽝と猫が⼀緒 にいる画像 「猫」クラスの 予測で重要 「⽝」クラスの 予測で重要 [Selvaraju+, 16]

Grad-CAM, Grad-CAM++ 31 DNNに対して予測への影響が⼤きい箇所を可視化 [Chattopadhyay+, 17]

Grad-CAM, Grad-CAM++ 32 DNNに対して予測への影響が⼤きい箇所を可視化 [Chattopadhyay+, 17] 2. 最終層をマップに利⽤ 1. 学習済みモデルを⽤意 k : channel（フィルタ数） i,j : ピクセルの位置 3. 各フィルタに重みをつける 4. ⾜し上げる Grad-CAM : 微分をピクセル内平均 Grad-CAM++ : 2, 3階微分も考慮

Grad-CAM : Python実装 33 • Keras Visualization Toolkit (Keras-vis)に搭載 • Keras : 深層学習のライブラリの⼀つ ü TensorflowやTheano上で動く（ラッパー） • ⽐較的簡単にDNNの実装ができる！ • 例 : バグとロシアンブルーの分類と可視化 https://github.com/raghakot/keras-vis http://marubon-ds.blogspot.com/2018/03/object-detection-by-cam-with-keras.html

Grad-CAM : Python実装 34 • 転移学習で2値分類のモデルを作成 • ImageNetを学習したVGG16の重みを利⽤ • 最終層だけ変えて学習 https://github.com/raghakot/keras-vis

Grad-CAM : Python実装 35 • Grad-CAMでヒートマップ作成 • visualize_cam()で評価 https://github.com/raghakot/keras-vis もとの図とヒートマップの⽐較するための関数を⽤意

Grad-CAM : Python実装 36 • Grad-CAMでヒートマップ作成 • クラス予測の寄与が⼤きい部分がより⾚くなる • モデルがうまく構築できているかの確認に使える https://github.com/raghakot/keras-vis 顔のなかでも⼝の部分 が予測に貢献している 顔のなかでも⽿の部分 が予測に貢献している

37 Summary

Summary 38 • 機械学習モデルと結果を解釈する⽅法を紹介 • どの特徴量が重要か • 各特徴量が予測にどう影響するか • 予測に対して特徴量がどう寄与するか • 研究、道具⽴ては進んできている • が、どういった場合に有効/使えないといった実際の 知⾒はあまりない • 今後はこれらを活⽤して実務側 → 研究側へのフィー ドバックをすることが重要になると考えている

39 Appendix

機械学習モデルと結果を解釈する⽅法 40 1. どの特徴量が重要か p Feature Importance 2. 各特徴量が予測にどう影響するか p Partial Dependence p Surrogate Model : Appendix 3. ある予測結果に対して特徴量がどう寄与するか p LIME, SHAP p Grad-CAM, Grad-CAM++ ⼤きくは、以下の3つの説明ができること （モデルの説明 : 1, 2 結果の説明 : 3）

Partial Dependence : Python実装 41 • PDPboxが複数のモデルに対応したライブラリ https://github.com/SauceCat/PDPbox

機械学習モデルと結果を解釈する⽅法 42 1. どの特徴量が重要か p Feature Importance 2. 各特徴量が予測にどう影響するか p Partial Dependence p Surrogate Model 3. ある予測結果に対して特徴量がどう寄与するか p LIME, SHAP p Grad-CAM, Grad-CAM++ ⼤きくは、以下の3つの説明ができること （モデルの説明 : 1, 2 結果の説明 : 3）

Surrogate Model 43 ブラックボックスモデルをまねる・代理モデルを作る x AAACZHichVHLSsNAFD2Nr1qrrRZBEKRYKq5kIoLiqujGZR/2AbWUJE5raJqEJC3W4g/oVnHhSkFE/Aw3/oCL/oAgLiu4ceFtGhAt6h1m5syZe+6cmZFNTbUdxjo+YWh4ZHTMPx6YCE5OhcLTMznbaFgKzyqGZlgFWbK5puo866iOxgumxaW6rPG8XNvu7eeb3LJVQ991WiYv1aWqrlZURXKISh2WwzG2wtyIDgLRAzF4kTTCt9jDPgwoaKAODh0OYQ0SbGpFiGAwiSuhTZxFSHX3OY4RIG2DsjhlSMTWaKzSquixOq17NW1XrdApGnWLlFHE2RO7Y132yO7ZC/v4tVbbrdHz0qJZ7mu5WQ6dzGXe/1XVaXZw8KX607ODCjZcryp5N12mdwulr28eXXQzm+l4e4lds1fyf8U67IFuoDfflJsUT18iQB8g/nzuQZBbXREJp9ZiiS3vK/yYxyKW6b3XkcAOksjSuRynOMO571kIChFhtp8q+DxNBN9CWPgE9k6J+A== AAACZHichVHLSsNAFD2Nr1qrrRZBEKRYKq5kIoLiqujGZR/2AbWUJE5raJqEJC3W4g/oVnHhSkFE/Aw3/oCL/oAgLiu4ceFtGhAt6h1m5syZe+6cmZFNTbUdxjo+YWh4ZHTMPx6YCE5OhcLTMznbaFgKzyqGZlgFWbK5puo866iOxgumxaW6rPG8XNvu7eeb3LJVQ991WiYv1aWqrlZURXKISh2WwzG2wtyIDgLRAzF4kTTCt9jDPgwoaKAODh0OYQ0SbGpFiGAwiSuhTZxFSHX3OY4RIG2DsjhlSMTWaKzSquixOq17NW1XrdApGnWLlFHE2RO7Y132yO7ZC/v4tVbbrdHz0qJZ7mu5WQ6dzGXe/1XVaXZw8KX607ODCjZcryp5N12mdwulr28eXXQzm+l4e4lds1fyf8U67IFuoDfflJsUT18iQB8g/nzuQZBbXREJp9ZiiS3vK/yYxyKW6b3XkcAOksjSuRynOMO571kIChFhtp8q+DxNBN9CWPgE9k6J+A== AAACZHichVHLSsNAFD2Nr1qrrRZBEKRYKq5kIoLiqujGZR/2AbWUJE5raJqEJC3W4g/oVnHhSkFE/Aw3/oCL/oAgLiu4ceFtGhAt6h1m5syZe+6cmZFNTbUdxjo+YWh4ZHTMPx6YCE5OhcLTMznbaFgKzyqGZlgFWbK5puo866iOxgumxaW6rPG8XNvu7eeb3LJVQ991WiYv1aWqrlZURXKISh2WwzG2wtyIDgLRAzF4kTTCt9jDPgwoaKAODh0OYQ0SbGpFiGAwiSuhTZxFSHX3OY4RIG2DsjhlSMTWaKzSquixOq17NW1XrdApGnWLlFHE2RO7Y132yO7ZC/v4tVbbrdHz0qJZ7mu5WQ6dzGXe/1XVaXZw8KX607ODCjZcryp5N12mdwulr28eXXQzm+l4e4lds1fyf8U67IFuoDfflJsUT18iQB8g/nzuQZBbXREJp9ZiiS3vK/yYxyKW6b3XkcAOksjSuRynOMO571kIChFhtp8q+DxNBN9CWPgE9k6J+A== AAACZHichVHLSsNAFD2Nr1qrrRZBEKRYKq5kIoLiqujGZR/2AbWUJE5raJqEJC3W4g/oVnHhSkFE/Aw3/oCL/oAgLiu4ceFtGhAt6h1m5syZe+6cmZFNTbUdxjo+YWh4ZHTMPx6YCE5OhcLTMznbaFgKzyqGZlgFWbK5puo866iOxgumxaW6rPG8XNvu7eeb3LJVQ991WiYv1aWqrlZURXKISh2WwzG2wtyIDgLRAzF4kTTCt9jDPgwoaKAODh0OYQ0SbGpFiGAwiSuhTZxFSHX3OY4RIG2DsjhlSMTWaKzSquixOq17NW1XrdApGnWLlFHE2RO7Y132yO7ZC/v4tVbbrdHz0qJZ7mu5WQ6dzGXe/1XVaXZw8KX607ODCjZcryp5N12mdwulr28eXXQzm+l4e4lds1fyf8U67IFuoDfflJsUT18iQB8g/nzuQZBbXREJp9ZiiS3vK/yYxyKW6b3XkcAOksjSuRynOMO571kIChFhtp8q+DxNBN9CWPgE9k6J+A== y AAACZHichVHLSsNAFD2Nr1qrrRZBEKRYKq7KVATFVdGNyz7sA2opSZzW0DQJSVqoxR/QreLClYKI+Blu/AEX/QFBXFZw48KbNCBa1DvMzJkz99w5MyMZqmLZjPV8wsjo2PiEfzIwFZyeCYVn5wqW3jJlnpd1VTdLkmhxVdF43lZslZcMk4tNSeVFqbHj7Bfb3LQUXduzOwavNMW6ptQUWbSJynSq4RhLMDeiwyDpgRi8SOvhW+zjADpktNAEhwabsAoRFrUykmAwiKugS5xJSHH3OY4RIG2LsjhliMQ2aKzTquyxGq2dmparlukUlbpJyiji7IndsT57ZPfshX38Wqvr1nC8dGiWBlpuVEMnC7n3f1VNmm0cfqn+9Gyjhk3Xq0LeDZdxbiEP9O2ji35uKxvvrrBr9kr+r1iPPdANtPabfJPh2UsE6AOSP597GBTWEknCmfVYatv7Cj8WsYxVeu8NpLCLNPJ0LscpznDuexaCQkSYH6QKPk8TwbcQlj4B+E6J+Q== AAACZHichVHLSsNAFD2Nr1qrrRZBEKRYKq7KVATFVdGNyz7sA2opSZzW0DQJSVqoxR/QreLClYKI+Blu/AEX/QFBXFZw48KbNCBa1DvMzJkz99w5MyMZqmLZjPV8wsjo2PiEfzIwFZyeCYVn5wqW3jJlnpd1VTdLkmhxVdF43lZslZcMk4tNSeVFqbHj7Bfb3LQUXduzOwavNMW6ptQUWbSJynSq4RhLMDeiwyDpgRi8SOvhW+zjADpktNAEhwabsAoRFrUykmAwiKugS5xJSHH3OY4RIG2LsjhliMQ2aKzTquyxGq2dmparlukUlbpJyiji7IndsT57ZPfshX38Wqvr1nC8dGiWBlpuVEMnC7n3f1VNmm0cfqn+9Gyjhk3Xq0LeDZdxbiEP9O2ji35uKxvvrrBr9kr+r1iPPdANtPabfJPh2UsE6AOSP597GBTWEknCmfVYatv7Cj8WsYxVeu8NpLCLNPJ0LscpznDuexaCQkSYH6QKPk8TwbcQlj4B+E6J+Q== AAACZHichVHLSsNAFD2Nr1qrrRZBEKRYKq7KVATFVdGNyz7sA2opSZzW0DQJSVqoxR/QreLClYKI+Blu/AEX/QFBXFZw48KbNCBa1DvMzJkz99w5MyMZqmLZjPV8wsjo2PiEfzIwFZyeCYVn5wqW3jJlnpd1VTdLkmhxVdF43lZslZcMk4tNSeVFqbHj7Bfb3LQUXduzOwavNMW6ptQUWbSJynSq4RhLMDeiwyDpgRi8SOvhW+zjADpktNAEhwabsAoRFrUykmAwiKugS5xJSHH3OY4RIG2LsjhliMQ2aKzTquyxGq2dmparlukUlbpJyiji7IndsT57ZPfshX38Wqvr1nC8dGiWBlpuVEMnC7n3f1VNmm0cfqn+9Gyjhk3Xq0LeDZdxbiEP9O2ji35uKxvvrrBr9kr+r1iPPdANtPabfJPh2UsE6AOSP597GBTWEknCmfVYatv7Cj8WsYxVeu8NpLCLNPJ0LscpznDuexaCQkSYH6QKPk8TwbcQlj4B+E6J+Q== AAACZHichVHLSsNAFD2Nr1qrrRZBEKRYKq7KVATFVdGNyz7sA2opSZzW0DQJSVqoxR/QreLClYKI+Blu/AEX/QFBXFZw48KbNCBa1DvMzJkz99w5MyMZqmLZjPV8wsjo2PiEfzIwFZyeCYVn5wqW3jJlnpd1VTdLkmhxVdF43lZslZcMk4tNSeVFqbHj7Bfb3LQUXduzOwavNMW6ptQUWbSJynSq4RhLMDeiwyDpgRi8SOvhW+zjADpktNAEhwabsAoRFrUykmAwiKugS5xJSHH3OY4RIG2LsjhliMQ2aKzTquyxGq2dmparlukUlbpJyiji7IndsT57ZPfshX38Wqvr1nC8dGiWBlpuVEMnC7n3f1VNmm0cfqn+9Gyjhk3Xq0LeDZdxbiEP9O2ji35uKxvvrrBr9kr+r1iPPdANtPabfJPh2UsE6AOSP597GBTWEknCmfVYatv7Cj8WsYxVeu8NpLCLNPJ0LscpznDuexaCQkSYH6QKPk8TwbcQlj4B+E6J+Q== x AAACZHichVHLSsNAFD2Nr1qrrRZBEKRYKq5kIoLiqujGZR/2AbWUJE5raJqEJC3W4g/oVnHhSkFE/Aw3/oCL/oAgLiu4ceFtGhAt6h1m5syZe+6cmZFNTbUdxjo+YWh4ZHTMPx6YCE5OhcLTMznbaFgKzyqGZlgFWbK5puo866iOxgumxaW6rPG8XNvu7eeb3LJVQ991WiYv1aWqrlZURXKISh2WwzG2wtyIDgLRAzF4kTTCt9jDPgwoaKAODh0OYQ0SbGpFiGAwiSuhTZxFSHX3OY4RIG2DsjhlSMTWaKzSquixOq17NW1XrdApGnWLlFHE2RO7Y132yO7ZC/v4tVbbrdHz0qJZ7mu5WQ6dzGXe/1XVaXZw8KX607ODCjZcryp5N12mdwulr28eXXQzm+l4e4lds1fyf8U67IFuoDfflJsUT18iQB8g/nzuQZBbXREJp9ZiiS3vK/yYxyKW6b3XkcAOksjSuRynOMO571kIChFhtp8q+DxNBN9CWPgE9k6J+A== AAACZHichVHLSsNAFD2Nr1qrrRZBEKRYKq5kIoLiqujGZR/2AbWUJE5raJqEJC3W4g/oVnHhSkFE/Aw3/oCL/oAgLiu4ceFtGhAt6h1m5syZe+6cmZFNTbUdxjo+YWh4ZHTMPx6YCE5OhcLTMznbaFgKzyqGZlgFWbK5puo866iOxgumxaW6rPG8XNvu7eeb3LJVQ991WiYv1aWqrlZURXKISh2WwzG2wtyIDgLRAzF4kTTCt9jDPgwoaKAODh0OYQ0SbGpFiGAwiSuhTZxFSHX3OY4RIG2DsjhlSMTWaKzSquixOq17NW1XrdApGnWLlFHE2RO7Y132yO7ZC/v4tVbbrdHz0qJZ7mu5WQ6dzGXe/1XVaXZw8KX607ODCjZcryp5N12mdwulr28eXXQzm+l4e4lds1fyf8U67IFuoDfflJsUT18iQB8g/nzuQZBbXREJp9ZiiS3vK/yYxyKW6b3XkcAOksjSuRynOMO571kIChFhtp8q+DxNBN9CWPgE9k6J+A== AAACZHichVHLSsNAFD2Nr1qrrRZBEKRYKq5kIoLiqujGZR/2AbWUJE5raJqEJC3W4g/oVnHhSkFE/Aw3/oCL/oAgLiu4ceFtGhAt6h1m5syZe+6cmZFNTbUdxjo+YWh4ZHTMPx6YCE5OhcLTMznbaFgKzyqGZlgFWbK5puo866iOxgumxaW6rPG8XNvu7eeb3LJVQ991WiYv1aWqrlZURXKISh2WwzG2wtyIDgLRAzF4kTTCt9jDPgwoaKAODh0OYQ0SbGpFiGAwiSuhTZxFSHX3OY4RIG2DsjhlSMTWaKzSquixOq17NW1XrdApGnWLlFHE2RO7Y132yO7ZC/v4tVbbrdHz0qJZ7mu5WQ6dzGXe/1XVaXZw8KX607ODCjZcryp5N12mdwulr28eXXQzm+l4e4lds1fyf8U67IFuoDfflJsUT18iQB8g/nzuQZBbXREJp9ZiiS3vK/yYxyKW6b3XkcAOksjSuRynOMO571kIChFhtp8q+DxNBN9CWPgE9k6J+A== AAACZHichVHLSsNAFD2Nr1qrrRZBEKRYKq5kIoLiqujGZR/2AbWUJE5raJqEJC3W4g/oVnHhSkFE/Aw3/oCL/oAgLiu4ceFtGhAt6h1m5syZe+6cmZFNTbUdxjo+YWh4ZHTMPx6YCE5OhcLTMznbaFgKzyqGZlgFWbK5puo866iOxgumxaW6rPG8XNvu7eeb3LJVQ991WiYv1aWqrlZURXKISh2WwzG2wtyIDgLRAzF4kTTCt9jDPgwoaKAODh0OYQ0SbGpFiGAwiSuhTZxFSHX3OY4RIG2DsjhlSMTWaKzSquixOq17NW1XrdApGnWLlFHE2RO7Y132yO7ZC/v4tVbbrdHz0qJZ7mu5WQ6dzGXe/1XVaXZw8KX607ODCjZcryp5N12mdwulr28eXXQzm+l4e4lds1fyf8U67IFuoDfflJsUT18iQB8g/nzuQZBbXREJp9ZiiS3vK/yYxyKW6b3XkcAOksjSuRynOMO571kIChFhtp8q+DxNBN9CWPgE9k6J+A== x AAACZHichVHLSsNAFD2Nr1qrrRZBEKRYKq5kIoLiqujGZR/2AbWUJE5raJqEJC3W4g/oVnHhSkFE/Aw3/oCL/oAgLiu4ceFtGhAt6h1m5syZe+6cmZFNTbUdxjo+YWh4ZHTMPx6YCE5OhcLTMznbaFgKzyqGZlgFWbK5puo866iOxgumxaW6rPG8XNvu7eeb3LJVQ991WiYv1aWqrlZURXKISh2WwzG2wtyIDgLRAzF4kTTCt9jDPgwoaKAODh0OYQ0SbGpFiGAwiSuhTZxFSHX3OY4RIG2DsjhlSMTWaKzSquixOq17NW1XrdApGnWLlFHE2RO7Y132yO7ZC/v4tVbbrdHz0qJZ7mu5WQ6dzGXe/1XVaXZw8KX607ODCjZcryp5N12mdwulr28eXXQzm+l4e4lds1fyf8U67IFuoDfflJsUT18iQB8g/nzuQZBbXREJp9ZiiS3vK/yYxyKW6b3XkcAOksjSuRynOMO571kIChFhtp8q+DxNBN9CWPgE9k6J+A== AAACZHichVHLSsNAFD2Nr1qrrRZBEKRYKq5kIoLiqujGZR/2AbWUJE5raJqEJC3W4g/oVnHhSkFE/Aw3/oCL/oAgLiu4ceFtGhAt6h1m5syZe+6cmZFNTbUdxjo+YWh4ZHTMPx6YCE5OhcLTMznbaFgKzyqGZlgFWbK5puo866iOxgumxaW6rPG8XNvu7eeb3LJVQ991WiYv1aWqrlZURXKISh2WwzG2wtyIDgLRAzF4kTTCt9jDPgwoaKAODh0OYQ0SbGpFiGAwiSuhTZxFSHX3OY4RIG2DsjhlSMTWaKzSquixOq17NW1XrdApGnWLlFHE2RO7Y132yO7ZC/v4tVbbrdHz0qJZ7mu5WQ6dzGXe/1XVaXZw8KX607ODCjZcryp5N12mdwulr28eXXQzm+l4e4lds1fyf8U67IFuoDfflJsUT18iQB8g/nzuQZBbXREJp9ZiiS3vK/yYxyKW6b3XkcAOksjSuRynOMO571kIChFhtp8q+DxNBN9CWPgE9k6J+A== AAACZHichVHLSsNAFD2Nr1qrrRZBEKRYKq5kIoLiqujGZR/2AbWUJE5raJqEJC3W4g/oVnHhSkFE/Aw3/oCL/oAgLiu4ceFtGhAt6h1m5syZe+6cmZFNTbUdxjo+YWh4ZHTMPx6YCE5OhcLTMznbaFgKzyqGZlgFWbK5puo866iOxgumxaW6rPG8XNvu7eeb3LJVQ991WiYv1aWqrlZURXKISh2WwzG2wtyIDgLRAzF4kTTCt9jDPgwoaKAODh0OYQ0SbGpFiGAwiSuhTZxFSHX3OY4RIG2DsjhlSMTWaKzSquixOq17NW1XrdApGnWLlFHE2RO7Y132yO7ZC/v4tVbbrdHz0qJZ7mu5WQ6dzGXe/1XVaXZw8KX607ODCjZcryp5N12mdwulr28eXXQzm+l4e4lds1fyf8U67IFuoDfflJsUT18iQB8g/nzuQZBbXREJp9ZiiS3vK/yYxyKW6b3XkcAOksjSuRynOMO571kIChFhtp8q+DxNBN9CWPgE9k6J+A== AAACZHichVHLSsNAFD2Nr1qrrRZBEKRYKq5kIoLiqujGZR/2AbWUJE5raJqEJC3W4g/oVnHhSkFE/Aw3/oCL/oAgLiu4ceFtGhAt6h1m5syZe+6cmZFNTbUdxjo+YWh4ZHTMPx6YCE5OhcLTMznbaFgKzyqGZlgFWbK5puo866iOxgumxaW6rPG8XNvu7eeb3LJVQ991WiYv1aWqrlZURXKISh2WwzG2wtyIDgLRAzF4kTTCt9jDPgwoaKAODh0OYQ0SbGpFiGAwiSuhTZxFSHX3OY4RIG2DsjhlSMTWaKzSquixOq17NW1XrdApGnWLlFHE2RO7Y132yO7ZC/v4tVbbrdHz0qJZ7mu5WQ6dzGXe/1XVaXZw8KX607ODCjZcryp5N12mdwulr28eXXQzm+l4e4lds1fyf8U67IFuoDfflJsUT18iQB8g/nzuQZBbXREJp9ZiiS3vK/yYxyKW6b3XkcAOksjSuRynOMO571kIChFhtp8q+DxNBN9CWPgE9k6J+A== 近似 近似 観測 ブラックボックスモデル 代理モデル ˆ y AAACanichVHLSsNAFD2N7/qqulHcFGvFlUxEUFwV3bjswz6gSkniqINpEpJpoYb+gDtXgl0piIif4cYfcOEniO4quHHhbRoQFfUOM3PmzD13zszojik8ydhjROnp7esfGByKDo+Mjo3HJiYLnl1zDZ43bNN2S7rmcVNYPC+FNHnJcblW1U1e1I82O/vFOnc9YVvbsuHw3ap2YIl9YWiSqOLOoSb9RrMSS7AlFkT8J1BDkEAYaTt2jR3swYaBGqrgsCAJm9DgUStDBYND3C584lxCItjnaCJK2hplccrQiD2i8YBW5ZC1aN2p6QVqg04xqbukjCPJHtgNa7N7dsue2PuvtfygRsdLg2a9q+VOZfxkOvf2r6pKs8Thp+pPzxL7WAu8CvLuBEznFkZXXz8+a+fWs0l/gV2yZ/J/wR7ZHd3Aqr8aVxmebSFKH6B+f+6foLC8pBLOrCRSG+FXDGIWc1ik915FCltIIx+4O8U5WpEXZVKZUWa7qUok1EzhSyjzHwOUjMY= AAACanichVHLSsNAFD2N7/qqulHcFGvFlUxEUFwV3bjswz6gSkniqINpEpJpoYb+gDtXgl0piIif4cYfcOEniO4quHHhbRoQFfUOM3PmzD13zszojik8ydhjROnp7esfGByKDo+Mjo3HJiYLnl1zDZ43bNN2S7rmcVNYPC+FNHnJcblW1U1e1I82O/vFOnc9YVvbsuHw3ap2YIl9YWiSqOLOoSb9RrMSS7AlFkT8J1BDkEAYaTt2jR3swYaBGqrgsCAJm9DgUStDBYND3C584lxCItjnaCJK2hplccrQiD2i8YBW5ZC1aN2p6QVqg04xqbukjCPJHtgNa7N7dsue2PuvtfygRsdLg2a9q+VOZfxkOvf2r6pKs8Thp+pPzxL7WAu8CvLuBEznFkZXXz8+a+fWs0l/gV2yZ/J/wR7ZHd3Aqr8aVxmebSFKH6B+f+6foLC8pBLOrCRSG+FXDGIWc1ik915FCltIIx+4O8U5WpEXZVKZUWa7qUok1EzhSyjzHwOUjMY= AAACanichVHLSsNAFD2N7/qqulHcFGvFlUxEUFwV3bjswz6gSkniqINpEpJpoYb+gDtXgl0piIif4cYfcOEniO4quHHhbRoQFfUOM3PmzD13zszojik8ydhjROnp7esfGByKDo+Mjo3HJiYLnl1zDZ43bNN2S7rmcVNYPC+FNHnJcblW1U1e1I82O/vFOnc9YVvbsuHw3ap2YIl9YWiSqOLOoSb9RrMSS7AlFkT8J1BDkEAYaTt2jR3swYaBGqrgsCAJm9DgUStDBYND3C584lxCItjnaCJK2hplccrQiD2i8YBW5ZC1aN2p6QVqg04xqbukjCPJHtgNa7N7dsue2PuvtfygRsdLg2a9q+VOZfxkOvf2r6pKs8Thp+pPzxL7WAu8CvLuBEznFkZXXz8+a+fWs0l/gV2yZ/J/wR7ZHd3Aqr8aVxmebSFKH6B+f+6foLC8pBLOrCRSG+FXDGIWc1ik915FCltIIx+4O8U5WpEXZVKZUWa7qUok1EzhSyjzHwOUjMY= AAACanichVHLSsNAFD2N7/qqulHcFGvFlUxEUFwV3bjswz6gSkniqINpEpJpoYb+gDtXgl0piIif4cYfcOEniO4quHHhbRoQFfUOM3PmzD13zszojik8ydhjROnp7esfGByKDo+Mjo3HJiYLnl1zDZ43bNN2S7rmcVNYPC+FNHnJcblW1U1e1I82O/vFOnc9YVvbsuHw3ap2YIl9YWiSqOLOoSb9RrMSS7AlFkT8J1BDkEAYaTt2jR3swYaBGqrgsCAJm9DgUStDBYND3C584lxCItjnaCJK2hplccrQiD2i8YBW5ZC1aN2p6QVqg04xqbukjCPJHtgNa7N7dsue2PuvtfygRsdLg2a9q+VOZfxkOvf2r6pKs8Thp+pPzxL7WAu8CvLuBEznFkZXXz8+a+fWs0l/gV2yZ/J/wR7ZHd3Aqr8aVxmebSFKH6B+f+6foLC8pBLOrCRSG+FXDGIWc1ik915FCltIIx+4O8U5WpEXZVKZUWa7qUok1EzhSyjzHwOUjMY= ˜ y AAACbHichVHLSgMxFD0dX7U+Wh8LoQjF0uJKUhEUV0U3LtVaq7RFZsaoofNiJi3U0h9w68KFulAQET/DjT/gwk8QwY2CGxfeTgdEi3pDkpOTe25OEs0xhCcZewwpXd09vX3h/sjA4NBwNDYyuunZVVfned02bHdLUz1uCIvnpZAG33JcrpqawQtaZbm1X6hx1xO2tSHrDi+b6r4l9oSuSqK2S1IYu7xRb+7EkmyG+ZHoBJkAJBHEqh27Rgm7sKGjChMcFiRhAyo8akVkwOAQV0aDOJeQ8Pc5moiQtkpZnDJUYis07tOqGLAWrVs1PV+t0ykGdZeUCaTYA7thr+ye3bIn9vFrrYZfo+WlTrPW1nJnJ3o0kXv/V2XSLHHwpfrTs8QeFnyvgrw7PtO6hd7W1w5PXnOL66lGml2yZ/J/wR7ZHd3Aqr3pV2t8/RQR+oDMz+fuBJuzMxnCa3PJ7FLwFWHEMYVpeu95ZLGCVeTpXBPHOMN56EUZV+LKZDtVCQWaMXwLJf0J/sqNrw== AAACbHichVHLSgMxFD0dX7U+Wh8LoQjF0uJKUhEUV0U3LtVaq7RFZsaoofNiJi3U0h9w68KFulAQET/DjT/gwk8QwY2CGxfeTgdEi3pDkpOTe25OEs0xhCcZewwpXd09vX3h/sjA4NBwNDYyuunZVVfned02bHdLUz1uCIvnpZAG33JcrpqawQtaZbm1X6hx1xO2tSHrDi+b6r4l9oSuSqK2S1IYu7xRb+7EkmyG+ZHoBJkAJBHEqh27Rgm7sKGjChMcFiRhAyo8akVkwOAQV0aDOJeQ8Pc5moiQtkpZnDJUYis07tOqGLAWrVs1PV+t0ykGdZeUCaTYA7thr+ye3bIn9vFrrYZfo+WlTrPW1nJnJ3o0kXv/V2XSLHHwpfrTs8QeFnyvgrw7PtO6hd7W1w5PXnOL66lGml2yZ/J/wR7ZHd3Aqr3pV2t8/RQR+oDMz+fuBJuzMxnCa3PJ7FLwFWHEMYVpeu95ZLGCVeTpXBPHOMN56EUZV+LKZDtVCQWaMXwLJf0J/sqNrw== AAACbHichVHLSgMxFD0dX7U+Wh8LoQjF0uJKUhEUV0U3LtVaq7RFZsaoofNiJi3U0h9w68KFulAQET/DjT/gwk8QwY2CGxfeTgdEi3pDkpOTe25OEs0xhCcZewwpXd09vX3h/sjA4NBwNDYyuunZVVfned02bHdLUz1uCIvnpZAG33JcrpqawQtaZbm1X6hx1xO2tSHrDi+b6r4l9oSuSqK2S1IYu7xRb+7EkmyG+ZHoBJkAJBHEqh27Rgm7sKGjChMcFiRhAyo8akVkwOAQV0aDOJeQ8Pc5moiQtkpZnDJUYis07tOqGLAWrVs1PV+t0ykGdZeUCaTYA7thr+ye3bIn9vFrrYZfo+WlTrPW1nJnJ3o0kXv/V2XSLHHwpfrTs8QeFnyvgrw7PtO6hd7W1w5PXnOL66lGml2yZ/J/wR7ZHd3Aqr3pV2t8/RQR+oDMz+fuBJuzMxnCa3PJ7FLwFWHEMYVpeu95ZLGCVeTpXBPHOMN56EUZV+LKZDtVCQWaMXwLJf0J/sqNrw== AAACbHichVHLSgMxFD0dX7U+Wh8LoQjF0uJKUhEUV0U3LtVaq7RFZsaoofNiJi3U0h9w68KFulAQET/DjT/gwk8QwY2CGxfeTgdEi3pDkpOTe25OEs0xhCcZewwpXd09vX3h/sjA4NBwNDYyuunZVVfned02bHdLUz1uCIvnpZAG33JcrpqawQtaZbm1X6hx1xO2tSHrDi+b6r4l9oSuSqK2S1IYu7xRb+7EkmyG+ZHoBJkAJBHEqh27Rgm7sKGjChMcFiRhAyo8akVkwOAQV0aDOJeQ8Pc5moiQtkpZnDJUYis07tOqGLAWrVs1PV+t0ykGdZeUCaTYA7thr+ye3bIn9vFrrYZfo+WlTrPW1nJnJ3o0kXv/V2XSLHHwpfrTs8QeFnyvgrw7PtO6hd7W1w5PXnOL66lGml2yZ/J/wR7ZHd3Aqr3pV2t8/RQR+oDMz+fuBJuzMxnCa3PJ7FLwFWHEMYVpeu95ZLGCVeTpXBPHOMN56EUZV+LKZDtVCQWaMXwLJf0J/sqNrw==

Surrogate Modelの例： defragTrees 44 [Hara and Hayashi, 16] アンサンブルモデル → シンプルな⽊構造のモデル • 簡略化 = モデル選択

Surrogate Modelの例： defragTrees 45 [Hara and Hayashi, 16] 論⽂の⼯夫 • （⽊ベースの）アンサンブルモデルを確率モデルとみなす • Factorized Asymptotic Bayesian (FAB) 推定を使って近 似解を求める テスト誤差は抑えつつ 必要なルールの数も抑える（上限は⼿で指定）

defragTrees：Python実装 46 • データセット : 乳がんのデータセット • アンサンブルモデル : XGBoost https://github.com/sato9hara/defragTrees TestデータのAUCは0.96程度

defragTrees：Python実装 47 • Surrogate Model • ルール⽣成 (errorは10%くらい) https://github.com/sato9hara/defragTrees

defragTrees：Python実装 48 • Surrogate Model • ルール⽣成 (errorは10%くらい) https://github.com/sato9hara/defragTrees 精度をあまり落とさずにシンプルなモデルで表現できた （実際には結果が安定しない場合があるため、試⾏錯誤が必要）

機械学習モデルの結果を解釈する⽅法 49 1. どの特徴量が重要か p Feature Importance 2. 各特徴量が予測にどう影響するか p Partial Dependence p Surrogate Model : Appendix 3. ある予測結果に対して特徴量がどう寄与するか p LIME, SHAP p Grad-CAM, Grad-CAM++ ⼤きくは、以下の3つの説明ができること （モデルの説明 : 1, 2 結果の説明 : 3）

LIME, SHAP 50 [Lundberg, Lee, 17] • ゲーム理論 p お互いの選択しだいで利得が変わるゲーム p 最終的な利益への貢献度を公正にエージェントへ分配 • 機械学習の解釈性 p お互いの値しだいで出⼒が変わるモデル p 最終的な出⼒への貢献度を公正に変数へ分配 Shapley valueは協⼒ゲーム理論で知られている

SHAP : Python実装 51 • SHAP value • 複数の予測値の傾向の可視化も可能 • Feature ImportanceやPartial Dependenceに近い 使い⽅も可能 https://github.com/slundberg/shap 各データの|SHAP|の平均 特徴量の値の⼤きさとSHAP value