論文紹介：Efficient Architecture Search by Network Transformation [Cai et al., AAAI-2018]

環境情報学府
情報メディア環境学専攻情報メディア学コース
白川研究室修士2年
斉藤翔汰
2018年5月8日
H. Cai, T. Chen, W. Zhang, Y. Yu, and J. Wan
Efficient Architecture Search by
Network Transformation
Thirty-Second AAAI Conference
on Artificial Intelligence (AAAI-18), 2018.

View Slide

概要
2
• Net2Net：もとの学習済みNeural Network
(NN)の構造を拡大して再学習する手法
• 重み継承（パラメータ共有）付き構造探索
• NAS：NNの構造を出力するRNNを用意し，
強化学習(REINFORCE)で構造学習する手法
Net2Net + Neural Architecture Search
提案手法を構成する技術
[Chen et al., 2016] [Zoph & Le, 2017]

View Slide

Table 3: Test error rate (%) comparison with state-of-the-art architectures.
Model Depth Params C10+
human
designed
ResNet (He et al. 2016a)
ResNet (stochastic depth) (Huang et al. 2017)
Wide ResNet (Zagoruyko and Komodakis 2016)
ResNet (pre-activation) (He et al. 2016b)
DenseNet (L = 40, k = 12) (Huang et al. 2017)
DenseNet-BC (L = 100, k = 12) (Huang et al. 2017)
110
1202
16
28
1001
40
100
190
1.7M
10.2M
11.0M
36.5M
10.2M
1.0M
0.8M
25.6M
6.61
4.91
4.81
4.17
4.62
5.24
4.51
3.46
auto
designed
Large-Scale Evolution (250 GPUs)(Real et al. 2017)
NAS (predicting strides, 800 GPUs) (Zoph and Le 2017)
NAS (max pooling, 800 GPUs) (Zoph and Le 2017)
NAS (post-processing, 800 GPUs) (Zoph and Le 2017)
EAS (plain CNN, 5 GPUs)
-
20
39
39
20
5.4M
2.5M
7.1M
37.4M
23.4M
5.40
6.01
4.47
3.65
4.23
action. The result is reported in Figure 5, which shows that
the RL based meta-controller can effectively focus on the
right search direction, while the random search cannot (left
plot), and thus find high performance architectures more ef-
ficiently than random search.
Explore DenseNet Architecture Space
概要
3
• 5GPUs × 4日で，NAS (800GPUs)やLarge-
Scale Evolution(250GPUs, 進化計算)
[Real et al., 2017] に匹敵する構造を発見
Model
human
designed
Wide ResNet (Zagoruyko and Komodakis 20
Wide ResNet (Zagoruyko and Komodakis 20
DenseNet (L = 40, k = 12) (Huang et al. 201
DenseNet-BC (L = 100, k = 12) (Huang et a
DenseNet-BC (L = 190, k = 40) (Huang et a
auto
designed
Large-Scale Evolution (250 GPUs)(Real et al
NAS (predicting strides, 800 GPUs) (Zoph an
NAS (max pooling, 800 GPUs) (Zoph and Le
NAS (post-processing, 800 GPUs) (Zoph and
Figure 5: Comparison between RL based meta-controller
and random search on C10+.
Table 4: Test error rate (%) results of exploring DenseNet
architecture space with EAS.
Model Depth Params C10 C10+
DenseNet (L = 100, k = 24) 100 27.2M 5.83 3.74
DenseNet-BC (L = 250, k = 24) 250 15.3M 5.19 3.62
DenseNet-BC (L = 190, k = 40) 190 25.6M - 3.46
NAS (post-processing) 39 37.4M - 3.65
EAS (DenseNet on C10) 70 8.6M 4.66 -
EAS (DenseNet on C10+) 76 10.7M - 3.44
actio
the R
right
plot)
ficie
Exp
We
spac
the s
bottl
and
plyin
achi
best
havi
error
3.46
less
In th
ward
スキップ
結合なし
スキップ
結合あり

View Slide

Introduction
4
• NASやLarge-Scale Evolutionなどは数百の
GPUを要する（計算コストの弱点）
• 人間がチューニングする際には，成功した
構造をもとにして少しずつ変えていくはず
• 計算コストの制約があるのに，成功した
構造・重みを捨てるのは無駄！
成功した構造と重みを再利用すれば
効率的に構造探索ができるはず

View Slide

Introduction
5
• ネットワーク変換：レイヤーの挿入，
ユニット数の増加，スキップ結合の追加など
• この論文ではスキップ結合なしのCNNと
DenseNet[Huang et al., 2017]を基にした空間を探索
• どのようなネットワーク変換操作を取るべきかを
強化学習で探索
ネットワーク変換＋
強化学習でパラメータ空間を探索
Efficient Architecture Search (EAS)

View Slide

Related Work and Background
6
1990 ~ 2000
Neuro-Evolution
2010 ~ 2016
Bayesian Optimization
2016 ~
強化学習ベースの手法
進化計算で構造や重みを最適化
目的関数をガウス過程によって
推定し，ある基準に従って探索
構造を探索するエージェントを考え，
強化学習で方策を更新
NAS以外にQ-Learningを用いた手法
[Baker et al., 2017] が存在
現在

View Slide

• NASの枠組み：Recurrent NN + REINFORCE
7
Child Network 1
(CNN)
… Child Network N
(CNN)
Meta-Controller
(RNN)
①RNNが学習した生成確率P
に基づいて構造を複数サンプル
Reward 1
(Accuracy)
Reward N
(Accuracy)
…
②NNを学習して
評価値を算出
③評価値をもとに勾配を計算
④勾配を使ってControllerの
パラメータを更新
m： Child Networkの個数，T：ハイパーパラメータの個数
r✓
J
(
✓
)
⇡ 1
m
m
X
k=
1
T
X
t=
1
(
R(
T
)
k
b
)
r✓ log
P
(
a(
t
)
| a(
t
1):1
;
✓
)

View Slide

8
• Net2Netの枠組み：出力を保つ構造変換
z
x y
a b
z
y y’
b
x
a
Original Network Transformed Network (Wider)
Net2WiderNet
Operator
②重みをコピー
③重みを1/2
①コピー元のユニット
をランダム選択
変換後も出力zは変わらない
function-preserving transformation
④変換後のネットワークを学習

View Slide

9
• もうひとつのNet2Net：Net2DeeperNet
z
x y
a b
z
y’
b
x’
a
Original Network Transformed Network (Deeper)
Net2DeeperNet
Operator
①コピー元の層を
ランダム選択
変換後も出力zは変わらない
③変換後のネットワークを学習
y
x
②新しい重みを
単位行列で初期化

View Slide

Architecture Search by Net Transformation
10
• 提案手法の概要：
Encoder
Network
Bi-LSTM Bi-LSTM Bi-LSTM Bi-LSTM
CONV(32,3,1) POOL(2,2) CONV(64,5,1) POOL(2,2) FC(64)
Net2Wider
Actor Network
...
Layer Embedding
Update the network
Multiple
Actor Networks
Layer Embedding Layer Embedding Layer Embedding Layer Embedding
Net2Deeper
Actor Network
Figure 1: Overview of the RL based meta-controller in EAS,
which consists of an encoder network for encoding the ar-
chitecture and multiple separate actor networks for taking
network transformation actions.
Encoder
Network
Net2Wider
Actor
Probability of
Widening the Layer
Sigmoid
Classifier
Sigmoid
Classifier
Sigmoid
Classifier
Decision of
Widening the Layer
Bi-LSTM Bi-LSTM
Block
Index
Layer
Index
Filter
Size
Stride
Parameters of New Layer
if Applicable (CNN as example)

Initial State
Encoder
Network
Net2Deeper
Actor
Bi-LSTM
①層の情報を全結合NNで低次元特徴に変換
②Bi-directional LSTMで構造全体をEncoding
③出力にSigmoidを適用し，ユニットを増やすか決定
④Decoder Netを用いて，追加する層の構造を決定

View Slide

Actor Networks (Net2Wider Actor)
11
• Net2WiderNetをCNNに適用
• つまり，出力チャネル数を大きくしていく
• 記号の定義：
• 層目のカーネル
• 層目のフィルタの幅・高さ： ,
• 層目の入力・出力チャネル数： ,
• Net2Widerで拡張したカーネル：
• ただし
K
l
= (
kl
w
,
kl
h
,
fl
i
,
fl
o
)
AAAEzXicnVPPT9RAFH5gVVx/AHox8dK4YNCQzSweZE00JHrQGAMsLJBQ2LRlFup226btLmCtVxP/AQ8ejCYejP+DF2PiSU8e+BOMR0y8ePCb1wJR3DVxmrbfe/Pe9755M2MFrhPFQuz09R/Rjh47PnCicPLU6TODQ8NnFyK/HdqyZvuuHy5ZZiRdx5O12IlduRSE0mxZrly0mrfU/GJHhpHje/PxdiBXWua65zQc24zhqg9VDKuV3EvriZvqN/Sx5ipAPdlMx/UcbgA2MugcQD+9XB8qitJEZVJcreiHQbkkeBQpHzP+cN87MmiNfLKpTS2S5FEM7JJJEZ5lKpOgAL4VSugBPCaFsByOkZRSAflt+CWiTHib+K7DWs69HmzFGzGDjUou3hCZOo2KL+KN2BUfxVvxVfzsypUwh9Kzjb+V5cqgPvj0/NyPf2a18I9p4yCrp+aYGjTJWh1oD9ijVmFn+Z2Hz3bnrldHk0vilfgG/S/FjniPFXid7/brWVl9zuy3kZN1MQS6nyuYBp+ER1mqB1dQxeCYdahU9VJWrWyDxvd9/8to0tYhxszXvWcx5hvgUucg6hlpgS2b92Bv8t62uIKHE6V0VLm6sb8DFp6EvdxLvGotJlalo6YD7CKzF+P8H4whUMLePcaYu+FBcYBspbsXn8Mo5XO+8hetdzFn4F/EvMFVCnxfJPQa1OE9yTqvIrJ8C/YWY4tP+9o+p04jHDeSMxVwX/cupd4dLEyUyqJUnhXFqZv5zR2gC3SRxsB1jaboDs1QDUpe0Af6RJ+1aa2tPdIeZ6H9fXnOOfptaE9+AdZ7DNs=
AAAEhHicnVM9bxNBEB2HA4IJJBENEo2FE4QQsvZo+KgioIACkThxEilnRXfntVlyX7pbm4ST/0BaCgoqkCgQ/4GGhj9AkZ+AKINEQ8HbucOImBiJPd3dzNuZN29nd70kUJkW4qAydcI6eer09Jnq2ZnqufOzczPrWdxPfdny4yBONz03k4GKZEsrHcjNJJVu6AVyw9u5Z+Y3BjLNVByt6b1EtkO3F6mu8l0NaHl7ri4agkdt3LBLo07liOcrH8ihDsXkU59CkhSRhh2QSxmeLbJJUAKsTTk9BeJSCk9xjKQhVZHfBy4R5QLdwbcHb6tEI/iGN2MGH5UCvCkya7QoPot34lB8Eu/FF/HjWK6cOYyePfy9Ilcm27P7F1e//zMrxF/Tk99ZEzVr6tIt1qqgPWHErMIv8gfPXx6u3mku5lfEG/EV+l+LA/ERK4gG3/y3K7L5itnvI6foYgrrUangMfgkEOOZHlxDFYdjelBp6g1ZtfEduj7C/pfRpd0xxgI7vmca811wmXOQTYz0wFbMR/Cf8d6GXCHCiTI6mlzdGe2AhydnlHuJ16zFxapqqKlgB8icxLh2hDGFlTP6i1FzNyIoTpBtdE/iU2wN+Zy3/6L1IeYc/OuYd7hKle+LhF6HBrwnRedNRJHvwd9l2+PT3hlx1miB4xZKpiquq330co4b6zcatmjYK4Km6RJdpquguElL9ICWqQUBHdqnF9YF67a1VFzrqUp5v+fpj2Hd/Qm+LfJ8
AAAEwnicnVNNb9NAEJ2UACUUmnJC4mKRFhVURRsufEhISHAAIdQ2bdpKdYhsd5Oa+Eu2k7YY8wP4Axw4IJA4IP4DF4TECU4c+hMQxyJx4cDbsUlFQ4PEWrbfzM68eTu7awaOHcVC7BbGjhSPHjs+fqJ0cuLU6cny1MRK5PdCSzYs3/HDNdOIpGN7shHbsSPXglAarunIVbN7S82v9mUY2b63HO8EsukaHc9u25YRw9UqX9NNN7mXthIn1W5os90HAK1kK53TcrgJ2M6gvQ/99GKrXBFVwUMbBrUcVCgfC/5U4R3ptEE+WdQjlyR5FAM7ZFCEZ51qJCiAr0kJPYTHoBCWzTGSUiohvwe/RJQBbxffDqz13OvBVrwRM1io5OANkanRjPgi3og98VG8FV/Fz0O5EuZQenbwN7NcGbQmn55d+vHPLBf/mDb3s0ZqjqlNV1mrDe0Be9QqrCy//+jZ3tL1+kxyQbwS36D/pdgV77ECr//der0o68+Z/TZysi6GQPdzBfPgk/AoS/XgEqroHNOBSlUvZdXK1mlu4PtfRoO2hxgz3+E9izHfBpc6B9HISBNs2bwHe4v31uUKHk6U0lHn6vpgB0w8CXu5l3jVWgysSkNNG9hB5ijG5QOMIVDC3t+MMXfDg+IA2Ur3KD6bUcrnvPkXrXcxp+NfwbzOVUp8XyT06tTnPck6ryKyfBP2NmOTT/vGgFOjaY6bzplKuK+1g7dzGKxcrtZEtbYoaJzO0XmaBcUVukl3aIEaEPCCPtAn+lycL/aKj7ObPVbIr/gZ+mMUn/wCwDQLSw==
AAAEwnicnVNNb9NAEJ2UACUUmnJC4mKRFhVURZtyoEFCQoIDCKG2adNWqkNku5vUxF+ynbTFmB/AH+DAAYHEAfEfuCAkTnDi0J+AOBaJCwfejtNWUBIk1rL9ZnbmzdvZXTNw7CgWYjc3cix//MTJ0VOF02Nnzo4XJ8ZWIr8bWrJu+Y4frplGJB3bk/XYjh25FoTScE1HrpqdW2p+tSfDyPa95XgnkA3XaHt2y7aMGK5msaqbbnIvbSZOqt3QpjsPAJrJVjqj9eEmYCuD9iH008vNYkmUZ6tz4mpVOwoqZcGjRP2x4E/k3pFOG+STRV1ySZJHMbBDBkV41qlCggL4GpTQQ3gMCmHZHCMppQLyu/BLRBnwdvBtw1rvez3YijdiBguVHLwhMjWaEl/EG7EnPoq34qv4OZArYQ6lZwd/M8uVQXP86fmlH//McvGPafMwa6jmmFo0x1ptaA/Yo1ZhZfm9R8/2lq7XppJL4pX4Bv0vxa54jxV4ve/W60VZe87st5GTdTEEut9XMA8+CY+yVA+uoIrOMW2oVPVSVq1snWYOfP/LaND2EcbMN7hnMeZb4FLnIBoaaYItm/dgb/HeulzBw4lSOmpcXT/YARNPwl7uJV61FgOr0lDTBnaQOYxx+Q/GEChh7z5jzN3woDhAttI9jM9mlPI5b/xF613M6fiXMK9zlQLfFwm9OvV4T7LOq4gs34S9zdjk075xwKnRJMdN9pkKuK/7l1IbDFZmyxVRriwKGqULdJGmQXGNbtIdWqA6BLygD/SJPufn89384+xmj+T6V/wc/TbyT34BOv0LqA==
l
AAAEk3icnVNNb9NAEJ0UAyV8tAUhIXGJSIsQQtGmPbQFCVXQAxwqmrRpK9VRZbubsNRfsp3QYuUPwBXEgRNIHBD/gQsX/gCH/gTEsUhcOPB27AZBSZBYy/bM25k3b2d37dBVcSLEfmHkmHH8xMnRU8XTZ86eGxufOL8WB53IkQ0ncINow7Zi6SpfNhKVuHIjjKTl2a5ct3fu6vn1roxiFfiryV4om57V9lVLOVYCqOZujZdFZXp+TszMl44a1YrgUaZ8LAcThQ9k0jYF5FCHPJLkUwLbJYtiPJtUJUEhsCal9AiIRRE8xTGSelREfge4RJQFdAffNrzNHPXha96YGRxUcvFGyCzRlPgs3okD8Um8F1/Ej4FcKXNoPXv421muDLfGnl5a+f7PLA//hB7+yhqqOaEWzbFWBe0hI3oVTpbfffLyYOVmfSq9Kt6Ir9D/WuyLj1iB3/3mvK3J+itmX0RO1sUI1lKu4AH4JBDt6R5cRxWTY9pQqev1WLX2TbrRx/6X0aLdI4wZNrhnCeZb4NLnIB4aaYMtm/fhP+a99biCjxOlddS5utnfARtPyij3Eq9ei4VVlVBTwXaROYxx9Q/GCFbK6CFjwt3woThEttY9jE+x1eNz3vyL1vuYM/EvY97kKkW+LxJ6TerynmSd1xFZvg1/l22bT/t2n7NEkxw3mTMVcV8PL2VpsLE2XamKSrUmygu385s7SpfpCl0D1ywt0D1apgaUSHpGz+mFcdG4ZdwxFrPQkUKec4F+G8bST0DN9Tc=
AAAEiHicnVM9bxNBEB0HA8EEkiAhIdFYOEEIIWuPJhE0CFJAgYidOImUs6K7y9osuS/drU3CyX8AWhAFFUgUiP9AQ8MfoMhPQJRBoqHg7dxhREyMxJ7ububtzJu3s7tu7KtUC7FfmjhWPn7i5OSpyumpM2enZ2an1tKol3iy5UV+lGy4Tip9FcqWVtqXG3EincD15bq7c8fMr/dlkqooXNV7sWwHTjdUHeU5GlDD35qpibrgUR01rMKoUTGWo9nSB7JpmyLyqEcBSQpJw/bJoRTPJlkkKAbWpoweAXEogac4RtKAKsjvAZeIcoDu4NuFt1mgIXzDmzKDh0o+3gSZVZoXn8U7cSA+iffii/hxJFfGHEbPHv5univjremnF1a+/zMrwF/Tw99ZYzVr6tAia1XQHjNiVuHl+f0nLw9WbjTns8vijfgK/a/FvviIFYT9b97bhmy+YvYl5ORdTGDdLxQ8AJ8EYjzTg6uoYnNMFypNvQGrNr5N14bY/zI6tDvCmGNH90xjvgMucw7SsZEu2PL5EP5j3tuAK4Q4UUZHk6vbwx1w8WSMci/xmrU4WFUVNRVsH5njGFcPMSawMkZ/MWruRgjFMbKN7nF8iq0Bn/P2X7Tew5yNfw3zNlep8H2R0GtTn/ck77yJyPNd+Ltsu3zat4ecVZrjuLmCqYL7ah2+naPG2vW6JepWQ9AkXaRLdAUUC3SL7tIytSBA0jN6Ti/K58s3y7fzmz1RKq74OfpjlJd+AiOQ8+E=
AAAEiHicnVNNb9NAEJ0UAyUU2iIhIXGxSIsQQtGmHGjhgqAHOCCatGkr1VFlu5uw1F+yndBi5Q/AFcSBE0gcEP+BCxf+AIf+BMSxSFw48HbsBkFJkLqW7Zm3M2/ezu46kaeSVIi90tgx4/iJk+OnyqcnzpydnJqeWE3CbuzKpht6Ybzu2In0VCCbqUo9uR7F0vYdT64523f1/FpPxokKg5V0N5It3+4Eqq1cOwVU9zanKqI6tzAvri+Yh41aVfCoUDGWwunSR7Joi0JyqUs+SQoohe2RTQmeDaqRoAhYizJ6DMSmGJ7iGEl9KiO/C1wiyga6jW8H3kaBBvA1b8IMLip5eGNkmjQrvoj3Yl98Fh/EV/FzKFfGHFrPLv5OniujzclnF5Z//DfLxz+lR7+zRmpOqU3zrFVBe8SIXoWb5/eevtpfvtmYzS6Lt+Ib9L8Re+ITVhD0vrvv6rLxmtkXkZN3MYb1oFDwEHwSiPZ0D66iisUxHajU9fqsWvsWXRtgR2W0aecQY44N71mK+Ta49DlIRkY6YMvnA/hPeG99rhDgRGkdDa5uDXbAwZMxyr3Eq9diY1UmairYHjJHMa78xRjDyhg9YEy5GwEUR8jWukfxKbb6fM5b/9B6H3MW/hXMW1ylzPdFQq9FPd6TvPM6Is934O+w7fBp3xpwmjTDcTMFUxn39eBSmsON1blqTVRrdUHjdJEu0RVQ3KDbdI+WqAkBkp7TC3ppnDduGXfymz1WKq74OfpjGIu/AJ5K9D4=
kl
w
AAAEm3icnVNNb9NAEJ0UAyV8tIULEkKKSIsQQtGmPdD2VKkcAIFo06YtqkNku5vgxl+ynbTF8h/gyoEDcACJA+I/cOHCH+DQn4A4FokLB96O3SAoCRJr2Z55O/Pm7eyuGTh2FAuxXxg5ph0/cXL0VPH0mbPnxsYnzq9Ffje0ZN3yHT/cMI1IOrYn67EdO3IjCKXhmo5cNzuLan69J8PI9r3VeC+QDddoe3bLtowY0MPOo8RJm8lO2hwvi8r03KyYmSsdNaoVwaNM+VjyJwofSKct8smiLrkkyaMYtkMGRXg2qUqCAmANSmgbiEEhPJtjJKVURH4XuESUAbSDbxveZo568BVvxAwWKjl4Q2SWaEp8Fu/Egfgk3osv4sdAroQ5lJ49/M0sVwbNsacXV77/M8vFP6bHv7KGao6pRbOs1Yb2gBG1CivL7z15frAyX5tKroo34iv0vxb74iNW4PW+WW+XZe0Fs99CTtbFENb9XMED8EkgylM9uI4qOse0oVLVS1m18nW60cf+l9Gg3SOMGTa4ZzHmW+BS5yAaGmmCLZv34O/w3rpcwcOJUjpqXF3v74CJJ2GUe4lXrcXAqkqoacN2kDmMcfUPxhBWwughY8zd8KA4QLbSPYzPZivlc974i9Y7mNPxL2Ne5ypFvi8SenXq8Z5knVcRWb4Jf5dtk0/7Vp+zRJMcN5kzFXFfDy9labCxNl2pikp1WZQX5vObO0qX6ApdA9dNWqDbtER1KHHpGb2kV9plbVG7q93LQkcKec4F+m1o9Z9v4PkS
AAAEkHicnVNNb9NAEJ2UACUU2nJCQkgRaRFCKFpzAXFCKgdAINq0aYvqENnuJizxl2wnbbH8B7hy4AAcQOKA+A9cuPAHOPQnII5F4sKBt2MTREOD1LVsz7ydefN2dtcOXRUnQuyWJo6Ujx47PnmicnLq1Onpmdmp1TjoR45sOoEbROu2FUtX+bKZqMSV62EkLc925ZrdW9DzawMZxSrwV5KdULY8q+urjnKsBNDD3qPUzdrpVtaeqYm64FEdNYzCqFExFoPZ0kcyaZMCcqhPHknyKYHtkkUxng0ySFAIrEUpPQFiUQRPcYykjCrI7wOXiLKA9vDtwtsoUB++5o2ZwUElF2+EzCrNiy/ivdgTn8UH8VX8PJArZQ6tZwd/O8+VYXv62dnlH//N8vBP6PGfrLGaE+rQddaqoD1kRK/CyfMHT1/sLd9ozKcXxVvxDfrfiF3xCSvwB9+dd0uy8ZLZbyEn72IE636h4AH4JBDt6R5cRhWTY7pQqetlrFr7Jl0ZYodltGh7hDHHDu5ZgvkOuPQ5iMdG2mDL5334W7y3HlfwcaK0jgZXN4c7YONJGeVe4tVrsbCqKmoq2C4yxzGu7GOMYKWM/mZMuBs+FIfI1rrH8Sm2Mj7nrX9ovYM5E/8a5k2uUuH7IqHXpAHvSd55HZHn2/C32bb5tG8OOas0x3FzBVMF99XYfztHjdWrdUPUjSVBk3SOLtAlUFyjm3SbFqkJAR49p1f0uny+vFC+m9/siVJxxc/QX6N87xcqsve4
AAAEkHicnVNNb9NAEJ0UAyUU2nJCQkgWaRFCKNqUAy0npHIABKJNm7aoDpHtboKJv2Q7aYvlP8CVAwfgABIHxH/gwoU/wKE/AXEsEhcOvB27QVASJNayPfN25s3b2V0rdJ04EWKvNHZEO3rs+PiJ8smJU6cnp6Yn1uKgF9myYQduEG1YZixdx5eNxElcuRFG0vQsV65b3UU1v96XUewE/mqyG8qmZ3Z8p+3YZgLoQfdh6matdDtrTVVEdW5hXlxd0A8btargUaFiLAXTpQ9k0BYFZFOPPJLkUwLbJZNiPJtUI0EhsCal9BiISRE8h2MkZVRGfg+4RJQJtItvB95mgfrwFW/MDDYquXgjZOo0Kz6Ld2JffBLvxRfxYyhXyhxKzy7+Vp4rw9bk07Mr3/+Z5eGf0KNfWSM1J9SmedbqQHvIiFqFnef3nzzfX7len00vijfiK/S/FnviI1bg97/Zb5dl/QWz30RO3sUI1r1CwX3wSSDKUz24jCoGx3SgUtXLWLXyDboywP6X0aSdQ4w5NrxnCebb4FLnIB4ZaYEtn/fhb/PeelzBx4lSOupc3RjsgIUnZZR7iVetxcSqdNR0YLvIHMW4+gdjBCtl9IAx4W74UBwiW+kexeewlfE5b/5F623MGfhXMG9wlTLfFwm9BvV5T/LOq4g834K/w7bFp31rwKnTDMfNFExl3NeDS6kPN9bmqjVRrS0LGqdzdIEugeIa3aBbtEQNCPDoGb2kV9p5bVG7k9/ssVJxxc/Qb0O7+xOlbPgV
AAAEkHicnVNNb9NAEJ2UACUU2nJCQkgRaRFCKFpzAXFCKgdAINq0aYvqENnuJizxl2wnbbH8B7hy4AAcQOKA+A9cuPAHOPQnII5F4sKBt2MTREOD1LVsz7ydefN2dtcOXRUnQuyWJo6Ujx47PnmicnLq1Onpmdmp1TjoR45sOoEbROu2FUtX+bKZqMSV62EkLc925ZrdW9DzawMZxSrwV5KdULY8q+urjnKsBNDD3qPUzdrpVtaeqYm64FEdNYzCqFExFoPZ0kcyaZMCcqhPHknyKYHtkkUxng0ySFAIrEUpPQFiUQRPcYykjCrI7wOXiLKA9vDtwtsoUB++5o2ZwUElF2+EzCrNiy/ivdgTn8UH8VX8PJArZQ6tZwd/O8+VYXv62dnlH//N8vBP6PGfrLGaE+rQddaqoD1kRK/CyfMHT1/sLd9ozKcXxVvxDfrfiF3xCSvwB9+dd0uy8ZLZbyEn72IE636h4AH4JBDt6R5cRhWTY7pQqetlrFr7Jl0ZYodltGh7hDHHDu5ZgvkOuPQ5iMdG2mDL5334W7y3HlfwcaK0jgZXN4c7YONJGeVe4tVrsbCqKmoq2C4yxzGu7GOMYKWM/mZMuBs+FIfI1rrH8Sm2Mj7nrX9ovYM5E/8a5k2uUuH7IqHXpAHvSd55HZHn2/C32bb5tG8OOas0x3FzBVMF99XYfztHjdWrdUPUjSVBk3SOLtAlUFyjm3SbFqkJAR49p1f0uny+vFC+m9/siVJxxc/QX6N87xcqsve4
AAAEkHicnVNNb9NAEJ0UAyUU2nJCQkgWaRFCKNqUAy0npHIABKJNm7aoDpHtboKJv2Q7aYvlP8CVAwfgABIHxH/gwoU/wKE/AXEsEhcOvB27QVASJNayPfN25s3b2V0rdJ04EWKvNHZEO3rs+PiJ8smJU6cnp6Yn1uKgF9myYQduEG1YZixdx5eNxElcuRFG0vQsV65b3UU1v96XUewE/mqyG8qmZ3Z8p+3YZgLoQfdh6matdDtrTVVEdW5hXlxd0A8btargUaFiLAXTpQ9k0BYFZFOPPJLkUwLbJZNiPJtUI0EhsCal9BiISRE8h2MkZVRGfg+4RJQJtItvB95mgfrwFW/MDDYquXgjZOo0Kz6Ld2JffBLvxRfxYyhXyhxKzy7+Vp4rw9bk07Mr3/+Z5eGf0KNfWSM1J9SmedbqQHvIiFqFnef3nzzfX7len00vijfiK/S/FnviI1bg97/Zb5dl/QWz30RO3sUI1r1CwX3wSSDKUz24jCoGx3SgUtXLWLXyDboywP6X0aSdQ4w5NrxnCebb4FLnIB4ZaYEtn/fhb/PeelzBx4lSOupc3RjsgIUnZZR7iVetxcSqdNR0YLvIHMW4+gdjBCtl9IAx4W74UBwiW+kexeewlfE5b/5F623MGfhXMG9wlTLfFwm9BvV5T/LOq4g834K/w7bFp31rwKnTDMfNFExl3NeDS6kPN9bmqjVRrS0LGqdzdIEugeIa3aBbtEQNCPDoGb2kV9p5bVG7k9/ssVJxxc/Qb0O7+xOlbPgV
l
kl
h
AAAEm3icnVNNb9NAEJ0UAyV8tIULEkKySIsQQtGmPfTjVKkcAIFo06YtqkNku5vUxF+yndBi5Q9w5cABOIDEAfEfuHDhD3DoT0Aci8SFA2/HbhCUBIm1bM+8nXnzdnbXCl0nToTYL4wc046fODl6qnj6zNlzY+MT59fjoBPZsmYHbhBtWmYsXceXtcRJXLkZRtL0LFduWO0lNb/RlVHsBP5ashfKume2fKfp2GYC6EH7Yer2GulOrzFeEuXp+TkxM68fNSplwaNE+VgOJgofyKBtCsimDnkkyacEtksmxXi2qEKCQmB1SukREJMieA7HSOpREfkd4BJRJtA2vi14Wznqw1e8MTPYqOTijZCp05T4LN6JA/FJvBdfxI+BXClzKD17+FtZrgwbY08vrn7/Z5aHf0I7v7KGak6oSXOs1YH2kBG1CjvL7z55frC6UJ1Kr4o34iv0vxb74iNW4He/2W9XZPUFs99ETtbFCNa9XMF98EkgylM9uI4qBse0oFLV67Fq5Rt0o4/9L6NJu0cYM2xwzxLMN8GlzkE8NNICWzbvw3/Me+txBR8nSumocnWjvwMWnpRR7iVetRYTq9JR04HtInMY49ofjBGslNFDxoS74UNxiGylexifw1aPz3n9L1pvY87Av4R5g6sU+b5I6DWoy3uSdV5FZPkW/F22LT7t231OnSY5bjJnKuK+Hl5KfbCxPl2uiHJlRZQWF/KbO0qX6ApdA9csLdItWqYalHj0jF7SK+2ytqTd0e5moSOFPOcC/Ta02k8xEPkD
l
fl
i
AAAEm3icnVNNb9NAEJ0UAyV8tIULEkKKSIsQQtGmHGh7qlQOgEC0adMW1Wlku5uw1F+yndBi5Q9w5cABOIDEAfEfuHDhD3DoT0Aci8SFA2/HbhCUBIm1bM+8nXnzdnbXDl0VJ0LsFUaOGEePHR89UTx56vSZsfGJs6tx0IkcWXcCN4jWbSuWrvJlPVGJK9fDSFqe7co1e3tBz691ZRSrwF9JdkPZ8Ky2r1rKsRJAD1qbqdtrpqrXHC+LyvTsjLg+WzpsVCuCR5nysRhMFD6QSVsUkEMd8kiSTwlslyyK8WxQlQSFwBqU0iMgFkXwFMdI6lER+R3gElEW0G182/A2ctSHr3ljZnBQycUbIbNEU+KzeCf2xSfxXnwRPwZypcyh9ezib2e5MmyOPT2//P2fWR7+CT38lTVUc0ItmmGtCtpDRvQqnCy/++T5/vJcbSq9LN6Ir9D/WuyJj1iB3/3mvF2StRfMfhM5WRcjWPdyBffBJ4FoT/fgKqqYHNOGSl2vx6q1b9K1Pva/jBbtHGLMsME9SzDfApc+B/HQSBts2bwP/zHvrccVfJworaPG1c3+Dth4Uka5l3j1WiysqoSaCraLzGGMK38wRrBSRg8YE+6GD8UhsrXuYXyKrR6f88ZftN7GnIl/GfMmVynyfZHQa1KX9yTrvI7I8m34O2zbfNq3+pwlmuS4yZypiPt6cClLg43V6UpVVKpLojw/l9/cUbpAl+gKuG7QPN2iRapDiUfP6CW9Mi4aC8Yd424WOlLIc87Rb8Oo/wQgLfj/
AAAEkHicnVNNb9NAEJ0UAyUU2nJCQkgRaRFCKFpzAXFCKgdAINq0aYvqENnuJiz1l2wntFj+A1w5cAAOIHFA/AcuXPgDHPoTEMciceHA27EJoqFBYi3bM29n3ryd3XUiTyWpELuViUPG4SNHJ49Vj0+dODk9Mzu1moT92JUtN/TCeN2xE+mpQLZSlXpyPYql7TueXHO2FvT82kDGiQqDlXQnkm3f7gWqq1w7BXS/+yDz8k6m8s5MXTQEj9qoYZZGncqxGM5WPpBFmxSSS33ySVJAKWyPbErwbJBJgiJgbcroERCbYniKYyTlVEV+H7hElA10C98evI0SDeBr3oQZXFTy8MbIrNG8+CzeiT3xSbwXX8SPA7ky5tB6dvB3ilwZdaafnl7+/s8sH/+UHv7OGqs5pS5dZa0K2iNG9CrcIn/w5Pne8rXmfHZevBFfof+12BUfsYJg8M19uySbL5j9BnKKLsaw7pYK7oFPAtGe7sFFVLE4pgeVul7OqrVv0aUh9r+MNm2PMBbYwT1LMd8Flz4HydhIB2zFfAD/Me+tzxUCnCito8nVreEOOHgyRrmXePVabKyqhpoKtofMcYwr+xhjWBmjvxhT7kYAxRGyte5xfIqtnM95+y9ab2HOwr+OeYurVPm+SOi1aMB7UnReRxT5Dvxtth0+7ZtDzhrNcdxcyVTFfTX3385RY/VywxQNc0nQJJ2hc3QBFFfoOt2kRWpBgE/P6CW9Ms4aC8bt4mZPVMorfor+GMadn9vB96U=
AAAEkHicnVNNb9NAEJ0UAyUU2nJCQkgWaRFCKNqUQ1tOSOUACESbNm1RHSLb3YRt/SXbCS1W/gBXDhyAA0gcEP+BCxf+AIf+BMSxSFw48HacBkFJKrGW7Zm3M2/ezu46kaeSVIi9wsgx4/iJk6OniqfHzpwdn5gcW03CduzKmht6Ybzu2In0VCBrqUo9uR7F0vYdT6452wt6fq0j40SFwUq6G8m6b7cC1VSunQJ62HyUed1GprqNiZIoz8zPievz5mGjUhY8StQbi+Fk4SNZtEkhudQmnyQFlML2yKYEzwZVSFAErE4ZbQGxKYanOEZSl4rIbwOXiLKBbuPbgrfRQwP4mjdhBheVPLwxMk2aFl/Ee7EvPosP4qv4OZArYw6tZxd/J8+VUWP82fnlH0dm+fin9Ph31lDNKTVpjrUqaI8Y0atw8/zO0xf7yzeq09ll8VZ8g/43Yk98wgqCznf33ZKsvmT2W8jJuxjDut9T8AB8Eoj2dA+uoorFMS2o1PW6rFr7Fl3rY//LaNPOIcYcG9yzFPNNcOlzkAyNdMCWzwfwn/De+lwhwInSOqpc3ervgIMnY5R7iVevxcaqTNRUsD1kDmNc+YsxhpUxesCYcjcCKI6QrXUP41Nsdfmc1/+h9Q7mLPxLmLe4SpHvi4Reizq8J3nndUSe78DfYdvh077Z5zRpiuOmekxF3NeDS2kONlZnyhVRriwJGqULdImugGKWbtJtWqQaBPj0nF7Ra+OisWDczW/2SKF3xc/RH8O49wtWivgC
AAAEkHicnVNNb9NAEJ0UAyUU2nJCQkgRaRFCKFpzAXFCKgdAINq0aYvqENnuJiz1l2wntFj+A1w5cAAOIHFA/AcuXPgDHPoTEMciceHA27EJoqFBYi3bM29n3ryd3XUiTyWpELuViUPG4SNHJ49Vj0+dODk9Mzu1moT92JUtN/TCeN2xE+mpQLZSlXpyPYql7TueXHO2FvT82kDGiQqDlXQnkm3f7gWqq1w7BXS/+yDz8k6m8s5MXTQEj9qoYZZGncqxGM5WPpBFmxSSS33ySVJAKWyPbErwbJBJgiJgbcroERCbYniKYyTlVEV+H7hElA10C98evI0SDeBr3oQZXFTy8MbIrNG8+CzeiT3xSbwXX8SPA7ky5tB6dvB3ilwZdaafnl7+/s8sH/+UHv7OGqs5pS5dZa0K2iNG9CrcIn/w5Pne8rXmfHZevBFfof+12BUfsYJg8M19uySbL5j9BnKKLsaw7pYK7oFPAtGe7sFFVLE4pgeVul7OqrVv0aUh9r+MNm2PMBbYwT1LMd8Flz4HydhIB2zFfAD/Me+tzxUCnCito8nVreEOOHgyRrmXePVabKyqhpoKtofMcYwr+xhjWBmjvxhT7kYAxRGyte5xfIqtnM95+y9ab2HOwr+OeYurVPm+SOi1aMB7UnReRxT5Dvxtth0+7ZtDzhrNcdxcyVTFfTX3385RY/VywxQNc0nQJJ2hc3QBFFfoOt2kRWpBgE/P6CW9Ms4aC8bt4mZPVMorfor+GMadn9vB96U=
AAAEkHicnVNNb9NAEJ0UAyUU2nJCQkgWaRFCKNqUQ1tOSOUACESbNm1RHSLb3YRt/SXbCS1W/gBXDhyAA0gcEP+BCxf+AIf+BMSxSFw48HacBkFJKrGW7Zm3M2/ezu46kaeSVIi9wsgx4/iJk6OniqfHzpwdn5gcW03CduzKmht6Ybzu2In0VCBrqUo9uR7F0vYdT6452wt6fq0j40SFwUq6G8m6b7cC1VSunQJ62HyUed1GprqNiZIoz8zPievz5mGjUhY8StQbi+Fk4SNZtEkhudQmnyQFlML2yKYEzwZVSFAErE4ZbQGxKYanOEZSl4rIbwOXiLKBbuPbgrfRQwP4mjdhBheVPLwxMk2aFl/Ee7EvPosP4qv4OZArYw6tZxd/J8+VUWP82fnlH0dm+fin9Ph31lDNKTVpjrUqaI8Y0atw8/zO0xf7yzeq09ll8VZ8g/43Yk98wgqCznf33ZKsvmT2W8jJuxjDut9T8AB8Eoj2dA+uoorFMS2o1PW6rFr7Fl3rY//LaNPOIcYcG9yzFPNNcOlzkAyNdMCWzwfwn/De+lwhwInSOqpc3ervgIMnY5R7iVevxcaqTNRUsD1kDmNc+YsxhpUxesCYcjcCKI6QrXUP41Nsdfmc1/+h9Q7mLPxLmLe4SpHvi4Reizq8J3nndUSe78DfYdvh077Z5zRpiuOmekxF3NeDS2kONlZnyhVRriwJGqULdImugGKWbtJtWqQaBPj0nF7Ra+OisWDczW/2SKF3xc/RH8O49wtWivgC
fl
o
AAAEm3icnVNNb9NAEJ0UAyV8tIULEkKySIsQQtGmHGh7qlQOgEC0adMW1Wlku5tg6i/ZTmix/Ae4cuAAHEDigPgPXLjwBzj0JyCOReLCgbdjNwhKgsRatmfezrx5O7trha4TJ0LslUaOaEePHR89UT556vSZsfGJs6tx0I1s2bADN4jWLTOWruPLRuIkrlwPI2l6livXrO0FNb/Wk1HsBP5KshvKpmd2fKft2GYC6EF7M3WzVhpkrfGKqE7Pzojrs/pho1YVPCpUjMVgovSBDNqigGzqkkeSfEpgu2RSjGeDaiQoBNaklB4BMSmC53CMpIzKyO8Cl4gygW7j24G3UaA+fMUbM4ONSi7eCJk6TYnP4p3YF5/Ee/FF/BjIlTKH0rOLv5XnyrA19vT88vd/Znn4J/TwV9ZQzQm1aYa1OtAeMqJWYef5vSfP95fn6lPpZfFGfIX+12JPfMQK/N43++2SrL9g9pvIybsYwbpXKLgPPglEeaoHV1HF4JgOVKp6GatWvkHX+tj/Mpq0c4gxxwb3LMF8G1zqHMRDIy2w5fM+/Me8tx5X8HGilI46Vzf6O2DhSRnlXuJVazGxKh01HdguMocxrvzBGMFKGT1gTLgbPhSHyFa6h/E5bGV8zpt/0Xobcwb+FcwbXKXM90VCr0E93pO88yoiz7fg77Bt8Wnf6nPqNMlxkwVTGff14FLqg43V6WpNVGtLojI/V9zcUbpAl+gKuG7QPN2iRWpAiUfP6CW90i5qC9od7W4eOlIqcs7Rb0Nr/AQ5TfkF
AAAEkHicnVNNb9NAEJ0UAyUU2nJCQkgRaRFCKFpzAXFCKgdAINq0aYvqENnuJiz1l2wntFj+A1w5cAAOIHFA/AcuXPgDHPoTEMciceHA27EJoqFBYi3bM29n3ryd3XUiTyWpELuViUPG4SNHJ49Vj0+dODk9Mzu1moT92JUtN/TCeN2xE+mpQLZSlXpyPYql7TueXHO2FvT82kDGiQqDlXQnkm3f7gWqq1w7BXS/+yDz8k4W5p2ZumgIHrVRwyyNOpVjMZytfCCLNikkl/rkk6SAUtge2ZTg2SCTBEXA2pTRIyA2xfAUx0jKqYr8PnCJKBvoFr49eBslGsDXvAkzuKjk4Y2RWaN58Vm8E3vik3gvvogfB3JlzKH17ODvFLky6kw/Pb38/Z9ZPv4pPfydNVZzSl26yloVtEeM6FW4Rf7gyfO95WvN+ey8eCO+Qv9rsSs+YgXB4Jv7dkk2XzD7DeQUXYxh3S0V3AOfBKI93YOLqGJxTA8qdb2cVWvfoktD7H8ZbdoeYSywg3uWYr4LLn0OkrGRDtiK+QD+Y95bnysEOFFaR5OrW8MdcPBkjHIv8eq12FhVDTUVbA+Z4xhX9jHGsDJGfzGm3I0AiiNka93j+BRbOZ/z9l+03sKchX8d8xZXqfJ9kdBr0YD3pOi8jijyHfjbbDt82jeHnDWa47i5kqmK+2ruv52jxurlhika5pKgSTpD5+gCKK7QdbpJi9SCAJ+e0Ut6ZZw1Fozbxc2eqJRX/BT9MYw7PwH0n/er
AAAEkHicnVNNb9NAEJ0UAyUU2nJCQkgWaRFCKNqUQ1tOSOUACESbNm1RHSLb3QS3/pLthBbLf4ArBw7AASQOiP/AhQt/gEN/AuJYJC4ceDt2g6AklVjL9szbmTdvZ3et0HXiRIi90sgx7fiJk6OnyqfHzpwdn5gcW42DbmTLhh24QbRumbF0HV82Eidx5XoYSdOzXLlmbS+o+bWejGIn8FeS3VA2PbPjO23HNhNAD9uPUjdrpUHWmqiI6sz8nLg+rx82alXBo0LFWAwmSx/JoE0KyKYueSTJpwS2SybFeDaoRoJCYE1KaQuISRE8h2MkZVRGfhe4RJQJdBvfDryNAvXhK96YGWxUcvFGyNRpWnwR78W++Cw+iK/i50CulDmUnl38rTxXhq3xZ+eXfxyZ5eGf0OPfWUM1J9SmOdbqQHvIiFqFnef3nr7YX75Rn04vi7fiG/S/EXviE1bg977b75Zk/SWz30JO3sUI1v1CwQPwSSDKUz24iioGx3SgUtXLWLXyDbrWx/6X0aSdQ4w5NrhnCebb4FLnIB4aaYEtn/fhP+G99biCjxOldNS5utHfAQtPyij3Eq9ai4lV6ajpwHaROYxx5S/GCFbK6AFjwt3woThEttI9jM9hK+Nz3vyH1juYM/CvYN7gKmW+LxJ6DerxnuSdVxF5vgV/h22LT/tmn1OnKY6bKpjKuK8Hl1IfbKzOVGuiWlsSNEoX6BJdAcUs3aTbtEgNCPDoOb2i19pFbUG7m9/skVJxxc/RH0O79wtvaPgI
ˆ
K
l
= (
kl
w
,
kl
h
,
fl
i
, ˆ
fl
o
)
AAAE2XicnVNNa9RQFL2tUev40aluBDfBaaVKGd7UhVNBKKigiNivaQtNHZL0TRsnXySZaWvIwp1068KFKwUX0v/gxo1bFy6qv0BcVnDjwvNu0orWjmBCknPPu/fc817es0LXiRMhdvr6j2hHjx0fOFE6eer0mcHy0Nn5OOhEtmzYgRtEi5YZS9fxZSNxElcuhpE0PcuVC1b7phpf6MoodgJ/LtkM5bJnrvpOy7HNBFSzfNtYM5PUsLz0XpY1UzfTb+ij7YcAzXQ9G9MLuAbYyqEDyEWtLCeC7HKzXBHV8Ym6uDqhHwS1quCrQsU1FQz1vSWDViggmzrkkSSfEmCXTIpxL1GNBIXglimlR2BMihA5nCMpoxLqO+AlskywbbxXES0VrI9Y6casYKOTiydCpU4j4qN4I3bFe7Etvogfh2qlrKH8bOJr5bUybA5unZ/9/s8qD9+E1n5V9fScUIvq7NWB95AZNQs7r+8+fr47e31mJL0kXomv8P9S7Ih3mIHf/Wa/npYzL1j9FmryVYyA7hcOHkBPglGRWoMr6GJwzipcqn4Zu1axQWP73P8qmrRxQDHnDl+zBOMtaKl9EPfMtKCWj/uI1/nfetzBx45SPma4u7H/ByzcKbO8lnjUXEzMSkdPB9hFZS/FuT8UI6CU2T3FhFfDh+MQ1cp3Lz2HUcb7fPkvXu9izMC3gnGDu5T4vEj4NajL/yRfeZWR11uINxhbvNtX9jV1Gua84UKphPO6dyj1w8H8eLUmqrVpUZmsFyd3gC7QRRqF1jWapDs0RQ042aYP9Ik+a0vaE+2ptpWn9vcVNefot0t79hNhKhJv
ˆ
fl
o
>
fl
o
AAAEr3icnVNNb9NAEJ0UAyV8NIULUi8WaRFCKNqUAykHVAkOcEC0adNUqtNgu5vU1LEt2wktlg9c+QMcOIGEEOICf4ALF/4Ah/4ExLFIXDjwdpwEQUkqsZbtmbczb97O7lqB60SxEPu5iWPa8RMnJ0/lT585e26qMH1+LfK7oS1rtu/64bplRtJ1PFmLndiV60EozY7lyrq1c1vN13syjBzfW433AtnomG3PaTm2GQNqFmaMbTNOWulm4qbNxE/1W3prYDcLRVGaX6iI6wv6YaNcEjyK1B9L/nTuIxm0RT7Z1KUOSfIohu2SSRGeDSqToABYgxJ6BMSkEJ7DMZJSyiO/C1wiygS6g28b3kYf9eAr3ogZbFRy8YbI1GlOfBFvxYH4LN6Jr+LnSK6EOZSePfytLFcGzalnF1d+HJnVwT+m7d9ZYzXH1KIKa3WgPWBErcLO8ntPnh+s3KzOJZfFK/EN+l+KffEJK/B63+3Xy7L6gtnvICfrYgjrfl/BA/BJIMpTPbiKKgbHtKFS1UtZtfINujbE/pfRpN1DjBk2umcx5lvgUucgGhtpgS2b9+A/5r3tcAUPJ0rpqHJ1Y7gDFp6EUe4lXrUWE6vSUdOB7SJzHOPqX4whrITRAWPM3fCgOEC20j2Oz2Er5XPe+IfWe5gz8C9i3uAqeb4vEnoN6vGeZJ1XEVm+BX+XbYtP+9aQU6dZjpvtM+VxXweXUh9trM2XyqJUXhbFxUr/5k7SDF2iK+C6QYt0l5aoBiVP6Q29pw9aWatrm9rDLHQi18+5QH8MzfkFjF0A9A==

View Slide

Gl( j)
=
8
>
>
<
>
>
:
j 1

j

fl
o
random sample from
{
1
, . . . ,
fl
o
}
fl
o
<
j

ˆ
fl
o
AAAFW3icnVPNThRBEK4FVnFEAY2JiZeOCwQJbnrxIBhNSDRRD0b+IaHXzcxs7zIwf8zMruBkXsAX8OBJEw/Gd/DiQV/AAwcfwHjExIsHv+mZBQVZE3sy01VfVX31dU+34dtWGHG+V+jp7SueOt1/Rjs7cO784NDwhZXQawWmXDY92wvWDD2UtuXK5ciKbLnmB1J3DFuuGlt30/hqWwah5blL0a4vq47edK2GZeoRoNrQR2HIpuXGcrulkIlEu1+L7WR88xq7w7Q8aqJDmGgMY5ONsQoTttyGqabGE+TXYi9hQqgU4RjeThzobt1zWKg7vi1ZI4CNjLgyyUTdi8LJ3+sSkB66tzvMYkOP4kbSCWhCuvWOFmUfqq4NlXh5amaa35hhx41KmatRonzMecOF9ySoTh6Z1CKHJLkUwbZJpxDPOlWIkw+sSjFtAtEpgGepHEkJaahvAZfI0oFu4duEt56jLvyUN1QMJjrZeANUMhrln/lbvs8/8Xf8K/95IlesOFI9u5iNrFb6tcHnlxd//LPKwRzRxmFVV80RNWhaabWg3VdIugozq28/e7G/eGthNB7jr/k36H/F9/gHrMBtfzffzMuFl4r9HmqyXQxgPcoVPAafBJJ66R5MoItQOU2oTPslSnXqC5o8wP6XUaedY4wZdvKeRYg3wJWeg7BrpgG2LO7Cf6r+raM6uDhRqY4F1V0c/AEDT6xQtZd407XoWBVDTwu2jcpujEtHGANYsUI7jJHaDReKfVSnurvxWcpK1Dmv/kXrQ8QE5hLiQnXR1H2R0Cuorf5JtvNpRlZvwN9RtqFOe/2Ak9GIyhvJmTTc186lZCcbK1PlCi9X5qdKs9P5ze2nK3SVxsF1k2bpAc3RMpmF64XFgihU+74Ue4tacSBL7SnkNRfpj1G89Askqzqx
AAAFUHicnVO9btRAEB4nOQgmkIQKicYiPwpRiNY0IAQSEkhAgch/ImWPk+3buzjxX2zfkWD5BXgBCiqQKBDvQEMBL0CRggdAlEGioeDz+C6BhBwSa9k7883MN9+ud+3Ic5NUiD2tr3+gcur04Bn97NC588Mjo0MrSdiKHbXshF4Yr9lWojw3UMupm3pqLYqV5dueWrW37hbx1baKEzcMltLdSFV9qxm4DdexUkC1kY/SVk03yNR2i5HpXL9fy7x8avOKcdvQO1EHHZJcNzA2jUnDNKSntmHy1HiC/FoW5oaUnCJ9O9zJYiuoh76RWH7kKaMRw0ZGZs4Ysh6myczvdTlID91bXWa5YaVZI+8GdKmCelcL24eqayNjYlbwMI4bZscYo86YC0e19ySpTiE51CKfFAWUwvbIogTPOpkkKAJWpYw2gVgUw3M5R1FOOupbwBWyLKBb+DbhrXfQAH7BmzCDg04e3hiVBk2Iz+Kt2BefxDvxVfw8kStjjkLPLma7rFVRbfj5xcUf/6zyMae0cVjVU3NKDbrBWl1ojxgpVuGU9e1nL/YXby5MZJPitfgG/a/EnviAFQTt786bebXwktnvoabcxRjWo46Cx+BTQAqv2INpdJGc04TKol/Oqgtf0swB9r+MFu0cYyyxk/csRbwBruIcJD0zbbCV8QD+U/63PncIcKIKHQvcXR78ARtPxijvJd5iLRZWZaCnC9tDZS/GpSOMMayM0S5jyrsRQHGE6kJ3Lz6XrZzPefUvWh8iJjGPIS65i873RUGvpDb/k3Lni4yy3oa/w7bNp71+wGnQOOeNd5h03Ffz6O08bqxcmzXFrDkvaJAu0WWaAsV1ukMPaI6WydGuaoua1KoDXyr9Fb282X1a54pfoD9GZegXBcQ5lg==
AAAFUHicnVPLbtNAFL1uGygm0JYVEpsRfahUJZqUBS0CCQkkYIHou5U6IbKdSerWr9pOSLH8A/wAC1YgsUD8AxsW8AMsuuADEMsisWHB8ThpgZIgMZY9955777lnxjNm4NhRzPmBNjA4VDh1eviMfrZ47vzI6FhxPfKboSXXLN/xw03TiKRje3IttmNHbgahNFzTkRvm7p0svtGSYWT73mq8H8iKazQ8u25bRgyoOvpBmLJhe4ncaypkJtXvVRMnnd65wm4xvRO10CFKdYaxw6ZYmQlH7sFUU/0x8quJnzIhVIpwTb+dhIZX810WGW7gSFYPYSMjKc8yUfPjaPbXuhSkx+7NLrPYNuKknnYDupBeratF2ceqq6PjvDS3MM+vLbCTRrnE1Rinzlj0x7R3JKhGPlnUJJckeRTDdsigCM8WlYlTAKxCCe0AMSiEZ6scSSnpqG8Cl8gygO7i24C31UE9+BlvpBgsdHLwhqhkNMk/8Tf8kH/kb/kX/qMnV6I4Mj37mM28VgbVkWcXV77/s8rFHNP2cVVfzTHVaV5ptaE9UEi2Ciuvbz19frhyY3kymeKv+Ffof8kP+HuswGt9s14vyeUXiv0uavJdDGE97Ch4BD4JJPOyPZhBF6FyGlCZ9UuV6swXNHuE/S+jQe0TjDnWe89ixOvgys5B1DfTBFse9+A/Uf/WVR08nKhMx7LqLo7+gIknUajaS7zZWgysiqGnDdtBZT/G1T8YQ1iJQruMsdoND4oDVGe6+/HZykrVOa/8ResDxATmccSF6qKr+yKhV1BL/ZN857OMvN6E31a2qU577YiT0YTKm+gw6biv3UvJehvrc6UyL5WXOA3TJbpM06C4TrfpPi3SGlnaVW1FE1pl6HNhsKDnN3tA61zxC/TbKBR/AoB+OfM=
AAAFW3icnVPNThRBEK4FVnFEAY2JiZeOCwQJbnrxIBhNSDRRD0b+IaHXzcxs7zIwf8zMruBkXsAX8OBJEw/Gd/DiQV/AAwcfwHjExIsHv+mZBQVZE3sy01VfVX31dU+34dtWGHG+V+jp7SueOt1/Rjs7cO784NDwhZXQawWmXDY92wvWDD2UtuXK5ciKbLnmB1J3DFuuGlt30/hqWwah5blL0a4vq47edK2GZeoRoNrQR2HIpuXGcrulkIlEu1+L7WR88xq7w7Q8aqJDmGgMY5ONsQoTttyGqabGE+TXYi9hQqgU4RjeThzobt1zWKg7vi1ZI4CNjLgyyUTdi8LJ3+sSkB66tzvMYkOP4kbSCWhCuvWOFmUfqq4NlXh5amaa35hhx41KmatRonzMecOF9ySoTh6Z1CKHJLkUwbZJpxDPOlWIkw+sSjFtAtEpgGepHEkJaahvAZfI0oFu4duEt56jLvyUN1QMJjrZeANUMhrln/lbvs8/8Xf8K/95IlesOFI9u5iNrFb6tcHnlxd//LPKwRzRxmFVV80RNWhaabWg3VdIugozq28/e7G/eGthNB7jr/k36H/F9/gHrMBtfzffzMuFl4r9HmqyXQxgPcoVPAafBJJ66R5MoItQOU2oTPslSnXqC5o8wP6XUaedY4wZdvKeRYg3wJWeg7BrpgG2LO7Cf6r+raM6uDhRqY4F1V0c/AEDT6xQtZd407XoWBVDTwu2jcpujEtHGANYsUI7jJHaDReKfVSnurvxWcpK1Dmv/kXrQ8QE5hLiQnXR1H2R0Cuorf5JtvNpRlZvwN9RtqFOe/2Ak9GIyhvJmTTc186lZCcbK1PlCi9X5nlpdjq/uf10ha7SOLhu0iw9oDlaJrNwvbBYEIVq35dib1ErDmSpPYW85iL9MYqXfgEkCzqv
• Net2WiderNetをrandom remapping
function で定式化：
• Net2WiderNet後のカーネル：
12
Gl
番目の出力チャンネル
について
元のCNNと同じ部分は再利用
新しく増えたチャネルは
既存のチャネルから
ランダムに選んで重みをコピー
j
AAAEk3icnVNNb9NAEJ0UAyV8tAUhIXGJSIsQQtGmHEjhUkEPcKho0qatVEeV7W6CW3/J3oQWK38AriAOnEDigPgPXLjwBzj0JyCOReLCgbdjNwhKgsRatmfezrx5O7trR56bKCH2C2PHjOMnTo6fKp4+c/bcxOTU+dUk7MaObDqhF8brtpVIzw1kU7nKk+tRLC3f9uSavXNPz6/1ZJy4YbCi9iLZ8q1O4LZdx1KA6tubk2VRmZ2riZtzpaNGtSJ4lCkfS+FU4QOZtEUhOdQlnyQFpGB7ZFGCZ4OqJCgC1qKUtoFYFMNzOUZSn4rI7wKXiLKA7uDbgbeRowF8zZswg4NKHt4YmSWaEZ/FO3EgPon34ov4MZQrZQ6tZw9/O8uV0ebE00vL3/+Z5eOv6NGvrJGaFbWpxlpdaI8Y0atwsvzek5cHy7cbM+lV8UZ8hf7XYl98xAqC3jfnbV02XjH7AnKyLsawFnMFD8EngWhP9+A6qpgc04FKXa/PqrVv0o0B9r+MFu0eYcyw4T1TmG+DS5+DZGSkDbZsPoD/mPfW5woBTpTW0eDq5mAHbDwpo9xLvHotFlZVQk0XtofMUYwrfzDGsFJGDxkVdyOA4gjZWvcoPpetPp/z1l+0PsCciX8Z8yZXKfJ9kdBrUo/3JOu8jsjybfi7bNt82rcGnCWa5rjpnKmI+3p4KUvDjdXZSlVUqnVRnq/lN3ecLtMVugauWzRP92mJmlAi6Rk9pxfGReOOcddYyELHCnnOBfptGIs/ATah9S8=
AAAEiHicnVM9bxNBEB2HA4IJJEFCQqKxcIIQQtaaBgQNghRQIGInTiLlrOjusnY2uS/drU3CyX8AWhAFFUgUiP9AQ8MfoMhPQJRBoqHg7dxhREyMxJ7ububtzJu3s7tu7KtUC7FfmjhmHT9xcvJU+fTUmbPTM7NTK2nUSzzZ8iI/StZcJ5W+CmVLK+3LtTiRTuD6ctXduWfmV/sySVUULuu9WLYDpxuqjvIcDaixvTFTFTXBozJq1AujSsVYjGZLH8imTYrIox4FJCkkDdsnh1I861QnQTGwNmW0DcShBJ7iGEkDKiO/B1wiygG6g28X3nqBhvANb8oMHir5eBNkVmhefBbvxIH4JN6LL+LHkVwZcxg9e/i7ea6MN6afXlj6/s+sAH9NW7+zxmrW1KGbrFVBe8yIWYWX5/efvDxYutWczy6LN+Ir9L8W++IjVhD2v3lvG7L5itkXkJN3MYH1sFDwCHwSiPFMD66iis0xXag09Qas2vg2XRti/8vo0O4IY44d3TON+Q64zDlIx0a6YMvnQ/iPeW8DrhDiRBkdTa5uD3fAxZMxyr3Ea9biYFUV1FSwfWSOY1w+xJjAyhj9xai5GyEUx8g2usfxKbYGfM7bf9H6AHM2/lXM21ylzPdFQq9Nfd6TvPMmIs934e+y7fJp3xxyVmiO4+YKpjLua/3w7Rw1Vq7X6qJWbwiapIt0ia6A4gbdofu0SC0IkPSMntML67x127qb3+yJUnHFz9Efw1r4CRtI898=
AAAEiHicnVNNb9NAEJ0UAyUU2iIhIXGxSIsQQtGmHGjhgqAHOCCatGkr1VFlu5uwrb9kO6HFyh+AK4gDJ5A4IP4DFy78AQ79CYhjkbhw4O3YDYKSILGW7Zm3M2/ezu46kaeSVIj90tgx4/iJk+OnyqcnzpydnJqeWE3CbuzKpht6Ybzu2In0VCCbqUo9uR7F0vYdT645O3f1/FpPxokKg5V0L5It3+4Eqq1cOwVU396cqojq3MK8uL5gHjVqVcGjQsVYCqdLH8iiLQrJpS75JCmgFLZHNiV4NqhGgiJgLcpoG4hNMTzFMZL6VEZ+F7hElA10B98OvI0CDeBr3oQZXFTy8MbINGlWfBbvxIH4JN6LL+LHUK6MObSePfydPFdGm5NPLyx//2eWj39Kj35ljdScUpvmWauC9ogRvQo3z+89eXmwfLMxm10Wb8RX6H8t9sVHrCDofXPf1mXjFbMvIifvYgzrQaHgIfgkEO3pHlxFFYtjOlCp6/VZtfYtujbA/pfRpt0jjDk2vGcp5tvg0ucgGRnpgC2fD+A/5r31uUKAE6V1NLi6NdgBB0/GKPcSr16LjVWZqKlge8gcxbjyB2MMK2P0kDHlbgRQHCFb6x7Fp9jq8zlv/UXrfcxZ+Fcwb3GVMt8XCb0W9XhP8s7riDzfgb/LtsOnfWvAadIMx80UTGXc18NLaQ43VueqNVGt1QWN00W6RFdAcYNu0z1aoiYESHpGz+mFcd64ZdzJb/ZYqbji5+i3YSz+BJYC9Dw=
ˆ
K
l
[
x
,
y
,
i
,
j
] = K
l
[
x
,
y
,
i
,
Gl
(
j
)]
AAAE7nicnVPdahNBFD6pq9b401RvBG8W00qVECb1wlZQCgoqIrZp0xa7IexuJ+km+8fuJiYueQEfQC9EREFQfAcRvPEFvOgLCOKFQgVvvPCb2W1E20Rwlt0955tzvvPNmRnDt60wYmw7M3ZAOXjo8PiR7NFjx09M5CZProZeOzB5xfRsL1g39JDblssrkRXZfN0PuO4YNl8zWtfE/FqHB6HluStRz+dVR2+4Vt0y9QhQLXdP29KjWDOc+Ha/X4vtvqrZRqCbLbVbUHsF1SqoTVULEuiKmgTuH3dDwDPN87vhtVyeFWfn59jFeXWvUSoyOfKUjkVvMvOWNNokj0xqk0OcXIpg26RTiGeDSsTIB1almJpAdArgWTKGU5+yyG8D54jSgbbwbcDbSFEXvuANJYOJSjbeAJkqTbOP7DXbYR/YG/aZ/RzKFUsOoaeHv5Hkcr828fD08o9/Zjn4R7T1O2uk5ojqNCe1WtDuS0SswkzyOw8e7yxfLk/H59gL9gX6n7Nt9h4rcDvfzZdLvPxEsl9HTtLFANadVMFd8HEgwhM9uIAqmoxpQKWo15eqha9RYYD9L6NO3T2MCTa8ZxHm6+AS5yAcGWmALZl34d+Xe+vICi5OlNBRltW1wQ4YeGKJyl7iFWvRsSoVNS3YNjJHMa78xRjAiiW6yxjJbrhQ7CNb6B7FZ0mrL895dR+ttzCn4Z/HvCarZOV94dCrUUfuSdJ5EZHkG/C70jbkad8ccKo0JeOmUqYs7uvupVSHG6uzxRIrlpZYfuFqenPH6QydpRlwXaIFukmLVIGSd/SJvtI3xVceKU+VZ0noWCbNOUV/DOXVL8ibF5g=
AAAE43icnVPNahNRFD6pUWusNnUluBlMK1VCuHGjCIKgoCJimzZtsRPCzPQmnWQyM8xMYuIwL+AD6EJEFFyI7yCCG1/ARV9AEBcKFdy48Ltn0oj9ScE7zMw53z3nO989917Td+wwEmIrM3Eke/TY8ckTuZNTp05P52emVkKvG1iyanmOF6yZRigd25XVyI4cueYH0uiYjlw12zfV/GpPBqHtucvRwJe1jtF07YZtGRGgev6hvmlEsW524ntJUo+dRNMdMzCsttYvaoOiZhe1lqYHKXRdSwP3j7ut4PnWxZ3wer4gSoKHttcoD40CDceCN5N5TzptkEcWdalDklyKYDtkUIhnncokyAdWo5haQAwK4NkcIymhHPK7wCWiDKBtfJvw1oeoC1/xhsxgoZKDN0CmRnPis3grtsUn8U58Fb8P5IqZQ+kZ4G+mudKvTz85u/Tr0KwO/hFt/s0aqzmiBl1lrTa0+4yoVVhpfu/xs+2la5W5+IJ4Lb5B/yuxJT5iBW7vp/VmUVaeM/st5KRdDGDdHyp4AD4JRHmqB5dQReeYJlSqegmrVr5OxRH2v4wG9fcwptjBPYsw3wCXOgfh2EgTbOm8C/8R722HK7g4UUpHhavrox0w8cSMci/xqrUYWJWGmjZsB5njGJd3MQawYkZ3GCPuhgvFPrKV7nF8NlsJn/PaPlrvYk7Hv4B5navk+L5I6NWpx3uSdl5FpPkm/D7bJp/2jRGnRrMcNztkyuG+lnffzr3GyuVSWZTKi4Im6Rydp3lQXKEbdIcWqAoBH+gLfacfWT/7NPsivdkTmeEVP0P/jOzLP0IjFec=
AAAE43icnVPdahNBFD6pUWusNvVK8GYwrVQJYVIvbAVBUFARsU2bttgNYXc7STfZP3Y3MXHJC/gAeiEiCl6I7yCCN76AF30BQbxQqOCNF34zm0RsmwjOsrvnfHPOd745M2P4thVGnO+mJo6kjx47Pnkic3Lq1Onp7MzUeui1AlOUTc/2gk1DD4VtuaIcWZEtNv1A6I5hiw2jeUPOb7RFEFqeuxZ1fVFx9Lpr1SxTjwBVsw+0HT2KNcOJ7/Z61djuMc02At1ssk6edfPMyrMG04IEusaSwMPjbkl4vnFxEF7N5nhhYWmRX15iB41igauRo/5Y9mZS70ijbfLIpBY5JMilCLZNOoV4tqhInHxgFYqpAUSnAJ6lYgT1KIP8FnCBKB1oE986vK0+6sKXvKFiMFHJxhsgk9Ec/8Tf8D3+kb/lX/ivkVyx4pB6uvgbSa7wq9OPz67+/GeWg39EO3+yxmqOqEaLSqsF7b5C5CrMJL/96One6tXSXHyBv+Jfof8l3+UfsAK3/cN8vSJKzxT7TeQkXQxg3esruA8+AUR6sgeXUEVTMXWolPV6SrX0NcoPsf9l1KlzgDHBRvcswnwNXPIchGMjDbAl8y78h2pvHVXBxYmSOkqqujbcAQNPrFDVS7xyLTpWxVDTgm0jcxzj2j7GAFas0AFjpLrhQrGPbKl7HJ+lrJ4655VDtN7BnIZ/DvOaqpJR90VAr0ZttSdJ52VEkm/A7yjbUKd9e8jJaFbFzfaZMrivg0vJRhvrC4UiLxRXOE3SOTpP86C4QtfpNi1TGQLe02f6Rt/TfvpJ+nlysydS/St+hv4a6Re/AbzdFkQ=
AAAE43icnVPNahNRFD6pUWusNnUluBlMK1VCuHGjCIKgoCJimzZtsRPCzPQmnWQyM8xMYuIwL+AD6EJEFFyI7yCCG1/ARV9AEBcKFdy48Ltn0oj9ScE7zMw53z3nO989917Td+wwEmIrM3Eke/TY8ckTuZNTp05P52emVkKvG1iyanmOF6yZRigd25XVyI4cueYH0uiYjlw12zfV/GpPBqHtucvRwJe1jtF07YZtGRGgev6hvmlEsW524ntJUo+dRNMdMzCsttYvaoOiZhe1lqYHKXRdSwP3j7ut4PnWxZ3wer4gSoKHttcoD40CDceCN5N5TzptkEcWdalDklyKYDtkUIhnncokyAdWo5haQAwK4NkcIymhHPK7wCWiDKBtfJvw1oeoC1/xhsxgoZKDN0CmRnPis3grtsUn8U58Fb8P5IqZQ+kZ4G+mudKvTz85u/Tr0KwO/hFt/s0aqzmiBl1lrTa0+4yoVVhpfu/xs+2la5W5+IJ4Lb5B/yuxJT5iBW7vp/VmUVaeM/st5KRdDGDdHyp4AD4JRHmqB5dQReeYJlSqegmrVr5OxRH2v4wG9fcwptjBPYsw3wCXOgfh2EgTbOm8C/8R722HK7g4UUpHhavrox0w8cSMci/xqrUYWJWGmjZsB5njGJd3MQawYkZ3GCPuhgvFPrKV7nF8NlsJn/PaPlrvYk7Hv4B5navk+L5I6NWpx3uSdl5FpPkm/D7bJp/2jRGnRrMcNztkyuG+lnffzr3GyuVSWZTKi4Im6Rydp3lQXKEbdIcWqAoBH+gLfacfWT/7NPsivdkTmeEVP0P/jOzLP0IjFec=
AAAE43icnVPdahNBFD6pUWusNvVK8GYwrVQJYVIvbAVBUFARsU2bttgNYXc7STfZP3Y3MXHJC/gAeiEiCl6I7yCCN76AF30BQbxQqOCNF34zm0RsmwjOsrvnfHPOd745M2P4thVGnO+mJo6kjx47Pnkic3Lq1Onp7MzUeui1AlOUTc/2gk1DD4VtuaIcWZEtNv1A6I5hiw2jeUPOb7RFEFqeuxZ1fVFx9Lpr1SxTjwBVsw+0HT2KNcOJ7/Z61djuMc02At1ssk6edfPMyrMG04IEusaSwMPjbkl4vnFxEF7N5nhhYWmRX15iB41igauRo/5Y9mZS70ijbfLIpBY5JMilCLZNOoV4tqhInHxgFYqpAUSnAJ6lYgT1KIP8FnCBKB1oE986vK0+6sKXvKFiMFHJxhsgk9Ec/8Tf8D3+kb/lX/ivkVyx4pB6uvgbSa7wq9OPz67+/GeWg39EO3+yxmqOqEaLSqsF7b5C5CrMJL/96One6tXSXHyBv+Jfof8l3+UfsAK3/cN8vSJKzxT7TeQkXQxg3esruA8+AUR6sgeXUEVTMXWolPV6SrX0NcoPsf9l1KlzgDHBRvcswnwNXPIchGMjDbAl8y78h2pvHVXBxYmSOkqqujbcAQNPrFDVS7xyLTpWxVDTgm0jcxzj2j7GAFas0AFjpLrhQrGPbKl7HJ+lrJ4655VDtN7BnIZ/DvOaqpJR90VAr0ZttSdJ52VEkm/A7yjbUKd9e8jJaFbFzfaZMrivg0vJRhvrC4UiLxRXOE3SOTpP86C4QtfpNi1TGQLe02f6Rt/TfvpJ+nlysydS/St+hv4a6Re/AbzdFkQ=

View Slide

13
• 出力を等しくする(function-preserving)
ため，出力側の重みに補正を入れる
• 層目のカーネルをとしたときの
Net2WiderNet適用後のカーネル：
l + 1
AAAElXicnVNNb9NAEJ0UAyV8tIUDSFws0iIEKFqXAy0CqRIVggNqmzRtpTqqbHcTTP0l2wktVv4AEld64AQSB8R/4MKFP8ChPwFxLBIXDrwdu0FQkkqsZXvm7cybt7O7duS5SSrEXmnkmHb8xMnRU+XTZ86eGxufOL+ShJ3YkQ0n9MJ4zbYS6bmBbKRu6sm1KJaWb3ty1d66r+ZXuzJO3DBYTnci2fStduC2XMdKAdW9G8bGeEVUp2dnxK1Z/bBhVAWPChVjMZwofSSTNikkhzrkk6SAUtgeWZTgWSeDBEXAmpTRUyAWxfBcjpHUozLyO8AloiygW/i24a0XaABf8SbM4KCShzdGpk5T4ot4L/bFZ/FBfBU/B3JlzKH07OBv57ky2hh7can+48gsH/+UnvzOGqo5pRbNsFYX2iNG1CqcPL/7fHe/fqc2lV0Vb8U36H8j9sQnrCDofnfeLcnaa2afR07exRjW40LBAvgkEOWpHlxHFZNj2lCp6vVYtfJNutnH/pfRou1DjDk2uGcp5lvgUucgGRppgy2fD+A/4731uUKAE6V01Li62d8BG0/GKPcSr1qLhVXpqOnC9pA5jHH5L8YYVsboAWPK3QigOEK20j2Mz2Wrx+e8+Q+tjzBn4l/BvMlVynxfJPSa1OU9yTuvIvJ8G/422zaf9s0+p06THDdZMJVxXw8upT7YWJmuGqJqLInK3N3i5o7SZbpC18B1m+boIS1SA0ra9JJe0a52UbunzWsP8tCRUpFzgf4Y2sIvEVj1pQ==
AAAEinicnVPPaxNBFH6pUWustvWi4CWYVkRLmPWiCIJgET2IbdK0hW4ou9tJHLu/2J3E1iX/gODVHjwpeBD/By9e/Ac89E8QjxW8ePCbt2vExkZwlt1975v3vvfNmxk39lWqhdgvTRwrHz9xcvJU5fTUmbPTM7NTq2nUSzzZ8iI/StZdJ5W+CmVLK+3L9TiRTuD6cs3dvmvm1/oySVUUrujdWLYDpxuqjvIcDajpX7M2Z2qiLnhURw2rMGpUjKVotvSBbNqiiDzqUUCSQtKwfXIoxbNBFgmKgbUpoydAHErgKY6RNKAK8nvAJaIcoNv4duFtFGgI3/CmzOChko83QWaV5sVn8U4ciE/ivfgifhzJlTGH0bOLv5vnynhz+vmF5vd/ZgX4a3r8O2usZk0duslaFbTHjJhVeHl+/9neQfNWYz67LN6Ir9D/WuyLj1hB2P/mvV2WjVfMvoicvIsJrIeFgkfgk0CMZ3pwFVVsjulCpak3YNXGt2lhiP0vo0M7I4w5dnTPNOY74DLnIB0b6YItnw/hP+W9DbhCiBNldDS4uj3cARdPxij3Eq9Zi4NVVVFTwfaROY5x5RBjAitj9Bej5m6EUBwj2+gex6fYGvA5b/9F6wPM2fjXMG9zlQrfFwm9NvV5T/LOm4g834W/w7bLp31ryFmlOY6bK5gquK/W4ds5aqxer1uibi0LmqSLdImugOIG3aH7tEQtCOjSC3pJe+Xz5dvlxfxmT5SKK36O/hjlez8B8DL0Tw==
AAAEinicnVNNb9NAEJ2UACUU2nIBiYtFWoQARetyoEVCQqJCcEC0SdNWqqPKdjdhqb9kO6HFyh9A4koPnEDigPgPXLjwBzj0JyCOReLCgbdjNwhKgsRatmfezrx5O7vrRJ5KUiH2S2PHysdPnBw/VTk9cebs5NT0xGoSdmNXNt3QC+N1x06kpwLZTFXqyfUolrbveHLN2b6r59d6Mk5UGKyku5Fs+XYnUG3l2imghnfN3Jyqitrcwry4sWAcNcya4FGlYiyF06UPZNEWheRSl3ySFFAK2yObEjwbZJKgCFiLMnoCxKYYnuIYSX2qIL8LXCLKBrqNbwfeRoEG8DVvwgwuKnl4Y2QaNCs+i3fiQHwS78UX8WMoV8YcWs8u/k6eK6PNyecXGt//meXjn9LjX1kjNafUpnnWqqA9YkSvws3ze8/2Dhq36rPZZfFGfIX+12JffMQKgt439+2yrL9i9kXk5F2MYT0sFDwCnwSiPd2Dq6hicUwHKnW9PqvWvkXXB9j/Mtq0c4Qxx4b3LMV8G1z6HCQjIx2w5fMB/Ke8tz5XCHCitI46V7cGO+DgyRjlXuLVa7GxKgM1FWwPmaMYV/5gjGFljB4yptyNAIojZGvdo/gUW30+562/aH2AOQv/KuYtrlLh+yKh16Ie70neeR2R5zvwd9h2+LRvDTgNmuG4mYKpgvt6eCmN4cbqXM0UNXNZ0DhdpEt0BRQ36Q7dpyVqQkCHXtBL2iufL98uL+Y3e6xUXPFz9Nso3/sJavv0rA==
Kl+1
AAAEnnicnVPNbtNAEJ4UAyX8tIULEheLtAgBijblQMsBVQIhEKrapk0bqY4i290EU//JdkKL5RfgAeDAgR+JA+IduHDhBTj0ERDHInHhwLdjNwhKgsRatme+nfnm29ldK3SdOBFirzR2RDt67Pj4ifLJU6fPTExOnV2Pg15ky4YduEHUtMxYuo4vG4mTuLIZRtL0LFduWNu31fxGX0axE/hryW4oW57Z9Z2OY5sJIMOwvPRB1k7dq7WsPVkR1dn5OXF9Xj9s1KqCR4WKsRxMlT6QQVsUkE098kiSTwlsl0yK8WxSjQSFwFqU0iMgJkXwHI6RlFEZ+T3gElEm0G18u/A2C9SHr3hjZrBRycUbIVOnGfFZvBP74pN4L76IH0O5UuZQenbxt/JcGbYnnp5f/f7PLA//hB7+yhqpOaEOzbFWB9pDRtQq7Dy//+T5/urN+kx6SbwRX6H/tdgTH7ECv//Nfrsi6y+Y/Q5y8i5GsBYLBUvgk0CUp3pwBVUMjulCpaqXsWrlG3RtgP0vo0k7hxhzbHjPEsx3wKXOQTwy0gJbPu/Df8x763EFHydK6ahzdWOwAxaelFHuJV61FhOr0lHTge0icxTj2h+MEayU0QPGhLvhQ3GIbKV7FJ/DVsbnvPUXrfcxZ+BfwbzBVcp8XyT0GtTnPck7ryLyfAv+DtsWn/atAadO0xw3XTCVcV8PLqU+3FifrdZEtbYiKgu3ips7ThfoIl0G1w1aoHu0TA0oCekZvaRXmq7d1Ra1pTx0rFTknKPfhtb8CVWO+cY=
AAAEk3icnVPNbtNAEJ6UACUU2nJC4mKRFiFA0ZoLiBMSCIFQ1TZt2kp1FHndTWrqP9lOaLH8AjwAHDjwI3FAvAMXLrwAhz4C4lgkLhz4dmyCaGiQWMv2zLcz33w7uysjz01SIfYrE8eqx0+cnDxVOz115uz0zOzUWhL2Y0e1nNAL4w1pJ8pzA9VK3dRTG1GsbF96al3u3NHz6wMVJ24YrKZ7kWr7di9wu65jp4AsS/rZw7yTeVfNvDNTFw3Bwxg1zNKoUzmWwtnKB7Joi0JyqE8+KQoohe2RTQmeTTJJUASsTRk9AmJTDM/lGEU51ZDfB64QZQPdwbcHb7NEA/iaN2EGB5U8vDEyDZoXn8U7cSA+iffii/hxJFfGHFrPHv6yyFVRZ/rp+ZXv/8zy8U9p+3fWWM0pdekma3WhPWJEr8Ip8gdPnh+s3GrOZ5fEG/EV+l+LffERKwgG35y3y6r5gtnvIqfoYgxroVSwCD4FRHu6B1dQxeKYHlTqejmr1r5F14bY/zLatDvCWGBH9yzFfBdc+hwkYyMl2Ir5AP5j3lufKwQ4UVpHk6tbwx2QeDJGuZd49VpsrMpATRe2h8xxjKuHGGNYGaO/GFPuRgDFEbK17nF8Lls5n/P2X7Q+wJyFfx3zFlep8X1R0GvRgPek6LyOKPIl/F22JZ/2rSGnQXMcN1cy1XBfzcO3c9RYu94wRcNcFjRJF+giXQbFDbpN92mJWhAQ0TN6Sa+qRvVedaG42ROV8oqfoz9GdfEnCCb4ZQ==
AAAEk3icnVPNbtNAEJ4UAyUU2nJC4mKRFiFA0bocaDkhgRAIVW3Tpq1UR5HtboKp/2Q7ocXyC/AAcODAj8QB8Q5cuPACHPoIiGORuHDg27EbBCVBYi3bM9/OfPPt7K4deW6SCrFfGTumHT9xcvxU9fTEmbOTU9MT60nYix3ZdEIvjDdtK5GeG8hm6qae3Ixiafm2JzfsndtqfqMv48QNg7V0L5It3+oGbsd1rBSQadp+9iBvZ95VI29P1UR9bmFeXF/QjxpGXfCoUTmWw+nKBzJpm0JyqEc+SQoohe2RRQmeLTJIUASsRRk9AmJRDM/lGEk5VZHfAy4RZQHdwbcLb6tEA/iKN2EGB5U8vDEydZoVn8U7cSA+iffii/gxlCtjDqVnD3+7yJVRe/Lp+dXv/8zy8U/p4a+skZpT6tA8a3WhPWJErcIp8vtPnh+s3mzMZpfEG/EV+l+LffERKwj635y3K7LxgtnvIKfoYgxrsVSwBD4JRHmqB1dQxeSYLlSqejmrVr5J1wbY/zJatHuEscCG9yzFfAdc6hwkIyNtsBXzAfzHvLc+VwhwopSOBlc3Bztg48kY5V7iVWuxsCodNV3YHjJHMa79wRjDyhg9ZEy5GwEUR8hWukfxuWzlfM5bf9F6H3Mm/jXMm1ylyvdFQq9Jfd6TovMqosi34e+ybfNp3x5w6jTDcTMlUxX39fBS6sON9bm6IerGiqBxukAX6TIobtAtukfL1ISAiJ7RS3ql6dpdbbG42WOV8oqfo9+GtvQTguD4wg==
ˆ
Kl+1
AAAEpnicnVPNbtNAEJ4UAyX8NC0XJC4RaVEFKNqUAy0HVAkOIAS0adNUqqNgu5vU1H+yndBi+QV4AQ5ISCAhQLwDFy68AIc+AuJYJC4c+HbsBkFJkFjL9sy3M998O7trBo4dxULsFcaOaEePHR8/UTx56vSZidLk1Frk90JLNizf8cN104ikY3uyEduxI9eDUBqu6cimuX1TzTf7Moxs31uNdwPZco2uZ3dsy4gBtUtT+pYRJ7rpJnfTtJ04l2tpu1QR1bmFeXF1oXzYqFUFjwrlY8mfLHwgnTbJJ4t65JIkj2LYDhkU4dmgGgkKgLUooUdADArh2RwjKaUi8nvAJaIMoNv4duFt5KgHX/FGzGChkoM3RGaZZsRn8U7si0/ivfgifgzlSphD6dnF38xyZdCeeHpu5fs/s1z8Y9r6lTVSc0wdmmetNrQHjKhVWFl+/8mz/ZXr9ZnkonglvkL/S7EnPmIFXv+b9XpZ1p8z+y3kZF0MYd3LFTwAnwSiPNWDS6iic0wXKlW9lFUrX6crA+x/GQ3aOcSYYcN7FmO+Ay51DqKRkSbYsnkP/mPeW5creDhRSkedq+uDHTDxJIxyL/GqtRhYVRk1bdgOMkcxrv7BGMJKGD1gjLkbHhQHyFa6R/HZbKV8zlt/0XoHczr+FczrXKXI90VCr0593pOs8yoiyzfh77Bt8mnfHHCWaZrjpnOmIu7rwaUsDzfW5qo1Ua0ti8rijfzmjtN5ukCz4LpGi3SblqgBJTv0gt7QW21Wu681tGYWOlbIc87Sb0N7+BPv4vzE
AAAEm3icnVPNbtNAEJ6UACUUmvaExCUiLaoARWsuIE5IcAAhoE2bplJdRba7SU39J3sTUiy/AC/AAQkJJASId+DChRfg0EdAHIvEhQPfjk0QDQ0Sa9me+Xbmm29nd+3IcxMlxH5p6lj5+ImT06cqp2fOnJ2tzs2sJ2E/dmTLCb0w3rCtRHpuIFvKVZ7ciGJp+bYn2/buLT3fHsg4ccNgTe1Fcsu3eoHbdR1LAepU580dS6Wm7af3sqyTepeNrFOti4bgURs3jMKoUzGWw7nSBzJpm0JyqE8+SQpIwfbIogTPJhkkKAK2RSk9AmJRDM/lGEkZVZDfBy4RZQHdxbcHb7NAA/iaN2EGB5U8vDEya7QoPot34kB8Eu/FF/HjSK6UObSePfztPFdGndmn51a//zPLx1/Rzu+siZoVdek6a3WhPWJEr8LJ8wdPnh2s3mguphfFK/EV+l+KffERKwgG35zXK7L5nNlvIyfvYgzrfqHgIfgkEO3pHlxCFZNjelCp62WsWvsmXRlh/8to0XCMMceO7pnCfBdc+hwkEyNtsOXzAfzHvLc+VwhworSOJlc3Rztg40kZ5V7i1WuxsKoaarqwPWROYlw7xBjDShn9xai4GwEUR8jWuifxuWxlfM63/qL1LuZM/OuYN7lKhe+LhF6TBrwneed1RJ5vwx+ybfNp3x5x1miB4xYKpgruq3H4do4b61cbhmgYK4Km6TxdoCVQXKObdIeWqQUBQ3pBb+htean8oNzKb/ZUqbji8/THKLd/AoMA+1s=
AAAEm3icnVPNbtNAEJ6UACUUmvaExCUiLaoARZtyoOWEBAcQAtq0aSrVUWS7m9TUf7I3IcXyC/ACHJCQQEKAeAcuXHgBDn0ExLFIXDjw7TgNgpIgdS3bM9/OfPPt7K4Vuk6shNjPTZzInzx1evJM4ezUufPTxZmpjTjoRras24EbRJuWGUvX8WVdOcqVm2EkTc9yZcPava3nGz0ZxU7gr6u9UDY9s+M7bcc2FaBWcdbYMVViWF5yP01biXu1mraKZVFZXF4S15dLR41qRfAo02CsBDO5j2TQNgVkU5c8kuSTgu2SSTGeLaqSoBBYkxJ6DMSkCJ7DMZJSKiC/C1wiygS6i28H3tYA9eFr3pgZbFRy8UbILNG8+CLeiwPxWXwQX8XPkVwJc2g9e/hbWa4MW9PPLqz9+G+Wh7+ind9ZYzUratMSa3WgPWREr8LO8ntPnx+s3azNJ5fFa/EN+l+JffEJK/B73+03q7L2gtnvICfrYgTrwUDBI/BJINrTPbiCKgbHdKBS10tZtfYNujbEjstoUv8IY4aN7pnCfBtc+hzEYyMtsGXzPvwnvLceV/BxorSOGlc3hjtg4UkY5V7i1WsxsaoSajqwXWSOY1z/izGClTB6yKi4Gz4Uh8jWusfxOWylfM6b/9B6D3MG/mXMG1ylwPdFQq9BPd6TrPM6Isu34PfZtvi0bw85SzTHcXMDpgLu6+GlLI02NhYrVVGprgqapIt0iRZAcYNu0V1aoToE9OklvaV3+YX8w3w9u9kTucEVn6U/Rr7xC/26+7g=
インデックスが重複した数で割る
ˆ
K
l
+1 = (
kl
+1
w
,
kl
+1
h
, ˆ
fl
+1
i
= ˆ
fl
o
,
fl
+1
o
)
AAAE9XicnVPNbtNAEJ4WAyX8NIULEheLtKhAFW3KgRaBVAkOIIRo06atVJfIdjeJqWNbtpO2WH4BXoAD4gACJMSBN+DCAV6AQx8BIU5F9MKBb8dJKigNEmvZnvl25ptvZ3etwHWiWIjtgcFD2uEjR4eO5Y6fOHlqOD9yejHyW6EtK7bv+uGyZUbSdTxZiZ3YlctBKM2m5cola/2mml9qyzByfG8h3grkatOse07Nsc0YUDVvGg0zTgyrmdxN02riXi6l+g19fP0Bm9VkI53Qe04DDsfX0i7kqPA9DIiPoFp32k8vVvMFUZycnhJXpvX9RqkoeBSoM2b9kYH3ZNAa+WRTi5okyaMYtksmRXhWqESCAmCrlNBDICaF8ByOkZRSDvkt4BJRJtB1fOvwVjqoB1/xRsxgo5KLN0SmTmPis3gjdsQn8VZ8ET8P5EqYQ+nZwt/KcmVQHX58dn73n1lN/GNq7GX11RxTjaZYqwPtASNqFXaW3370ZGf+WnksuSBeiK/Q/1xsiw9Ygdf+br+ak+WnzH4LOVkXQ1j3Ogrug08CUZ7qwSVUMTimDpWqXsqqlW/QRA/7X0aTNvcxZtjBPYsxXwOXOgdR30gLbNm8B3+D97bJFTycKKWjzNWN3g5YeBJGuZd41VpMrEpHTQe2i8x+jAt/MIawEka7jDF3w4PiANlKdz8+h62Uz/nqX7TewZyBfwHzBlfJ8X2R0GtQm/ck67yKyPIt+JtsW3za13qcOo1y3GiHKYf72r2U+sHG4mSxJIqlOVGYud65uUN0js7TOLiu0gzdplmqQMlH+kY/aFfb0J5pL7XXWejgQCfnDP02tHe/AKJUHFM=
ˆ
K
l
+1
[
x
,
y
,
j
,
k
] =
K
l
+1
[
x
,
y
,
Gl
(
j
),
k
]
#{
z
|
Gl
(
z
) =
Gl
(
j
)}
AAAFHXicnVNNa9RAGH63rlrXj7aKIAg6uK20uiyz9WArKAUFFRH73UKnLEk626abZEOSrt2G/AD9Ax48iIIH8T94EcGDeBPpTxAPHip48eAzk7RV213BhCTv+8z7PO8zkxnTd+ww4nwz13Ugf/DQ4e4jhaPHjp/o6e07ORs21gJLzlgNpxHMm0YoHduTM5EdOXLeD6Thmo6cM+s31fhcUwah3fCmo5YvF11j2bNrtmVEgKq9j8SKEcXCdON7SVKNncuVhAnHDAyrztZLrFViqyVWZyJIoetM1BBkhH3rbwNNBleHfqMlsSiKeIMJ117KxjeGoLVdykTCkmpvkZeHR0f4lVG2N6iUub6KlF3jjb7cGxK0RA2yaI1ckuRRhNghg0LcC1QhTj6wRYppFYhBATJb10hKqAD+GnCJKgNoHe9lZAsZ6iFXuqFWsNDJwROAyWiAf+Kv+BZ/z1/zL/xnW61Yayg/LXzNlCv9as/jM1M//sly8Y1oZZfV0XNENRrRXm149zWiZmGl/ObGk62pa5MD8UX+gn+F/+d8k7/FDLzmd+vlhJx8qtVvgZOuYoDofubgAfQkEJWpNbiELkLXLMOl6pdo1yoXVNrB/lfRoPU9iinWfs0ijNegpfZB2LHShFo67iF/qP+tqzt42FHKx6TuLnb+gIk71qheSzxqLgZmxdDTRuyA2Ulx+i/FAFGs0W3FSK+GB8c+2Mp3Jz1bR4ne54v7eL2LMYFvEeNCdyno8yLhV1BT/5N05VVFyjeRr+vY1Lt9aUeTUb+u68+UCjiv24eStQ9mh8sVXq5M8OLYjezkdtNZukCD0LpKY3SHxmkGTr7lTufO5c7nn+Xf5T/kP6alXbmMc4r+uPKffwGO1iXr
AAAFEnicnVNNa9RQFL1TR61jta0gCII+nFZaHYY3bhRBEBRURGynn9A3DEnmTZuZJBOSzNg25AfoH3DhQhRciP/BjQguxJ2L/gRx4aKCGxee95JW7bQj+EKSe8+999zzvkzfscOI863c0KH84SNHh48Vjo+cODk6Nj6yGHa6gSUXrI7TCZZNI5SO7cmFyI4cuewH0nBNRy6Z7VsqvtSTQWh3vPlow5c111j17KZtGRGg+thjsWZEsTDd+H6S1GPnciVhwjEDw2qz9RLbKLFWibWZCFLoBhNNGFnBvvl3gCZTrek/ypJYFEW8yYRrN7L45jS4dlKZSFhSHyvyMteD9RuVzChSNmY647m3JKhBHbKoSy5J8iiC7ZBBIZ4VqhAnH1iNYmoBMSiAZ+scSQkVUN8FLpFlAG3juwpvJUM9+Io31AwWOjl4A1QymuSf+Wu+zT/wN/wL/3kgV6w5lJ4N/M20Vvr10Sdn5n78s8rFP6K131UDNUfUpGtaqw3tvkbULKy0vrf5dHvuenUyvshf8q/Q/4Jv8XeYgdf7br2aldVnmv02atJVDGA9yBQ8BJ8Eojy1BpfQReicVahU/RKtWvmCSrvY/zIatN7HmGIHr1mEeBNc6hyEAzNNsKVxD/4jvbeu7uDhRCkdVd1d7O6AiSfWqF5LvGouBmbF0NOG7aByEOP8HsYAVqzRHcZIr4YHxT6qle5BfLa2En3Oa/tovYeYwL+IuNBdCvq+SOgV1NN7kq68ykjrTfjr2jb1aW/scjKa0HkTGVMB97Wy93b2G4tXyhVersxyGqazdIGmQHGVbtJdmqEFCPiWO507lzuff55/n/+Y3uyhXHbFT9FfI//pF1hLJAs=
AAAFEnicnVNNa9RQFL1TR61jta0gCII+nFZaHYY3dWErCIKCiojt9BP6hiFJ37TpJJmQpGPbkB+gf8CFC1FwIf4HNyK4EHcu+hPEhYsKblx43ktm/Gg7gi8kuffce88978v0HTuMON/J9R3KHz5ytP9Y4fjAiZODQ8MDC2FrI7DkvNVyWsGSaYTSsT05H9mRI5f8QBqu6chFs3lTxRfbMgjtljcXbfmy5hqrnt2wLSMCVB96JNaMKBamG99LknrsXK4kTDhmYFhNtlliWyW2XmJNJoIUus5EA0ZWsG/+baDJ2Pr4b2VJLIoi3mbCtVey+PY4uDqpTCQsqQ8VeXliapJfmWJ7jUqZ61GkbEy3hnNvSNAKtciiDXJJkkcRbIcMCvEsU4U4+cBqFNM6EIMCeLbOkZRQAfUbwCWyDKBNfFfhLWeoB1/xhprBQicHb4BKRqP8E3/Fd/l7/pp/5j8O5Io1h9Kzhb+Z1kq/Pvj4zOz3f1a5+Ee09quqp+aIGjSptdrQ7mtEzcJK69vbT3Znr1VH44v8Bf8C/c/5Dn+LGXjtb9bLGVl9qtlvoSZdxQDW/UzBA/BJIMpTa3AJXYTOWYVK1S/RqpUvqNTF/pfRoM09jCl28JpFiDfApc5B2DPTBFsa9+A/1Hvr6g4eTpTSUdXdRXcHTDyxRvVa4lVzMTArhp42bAeVvRjn/mIMYMUa7TBGejU8KPZRrXT34rO1lehzXttH613EBP5FxIXuUtD3RUKvoLbek3TlVUZab8Lf1LapT/tKl5PRiM4byZgKuK+dS8kONhYmyhVersxw6qezdIHGQHGVbtAdmqZ5CPiaO507lzuff5Z/l/+Q3uy+XHbFT9EfI//xJ9MFJGg=

View Slide

14
• Net2Wider Actorでは，Bi-directional
LSTMの出力に直接sigmoid関数を適用
• 出力チャネルを増加させるかどうか
• 「増加」の場合は
Net2WiderNetを実行
• 次の離散レベルへ
出力チャネル数を変更
• 例：32個→64個
Encoder
Network
Net2Wider
Actor
Probability of
Widening the Layer
Sigmoid
Classifier
Sigmoid
Classifier
Sigmoid
Classifier
Decision of
Widening the Layer
Bi-LSTM Bi-LSTM
Figure 2: Net2Wider actor, which uses a shared sigmoid
classiﬁer to simultaneously determine whether to widen
each layer based on its hidden state given by the encoder
network.

View Slide

Actor Networks (Net2Deeper Actor)
15
• Net2Deeper ActorではRNNを用いて
挿入する層の情報を出力
• 畳み込み層をどこに挿入するか，フィルタサイズ・
ストライドの大きさをどうするか
• グローバル平均プーリングや
全結合層の後に挿入する
場合は全結合層に限定
• 挿入された層の重みは
単位行列で初期化
Block
Index
Layer
Index
Filter
Size
Stride
Parameters of New Layer
if Applicable (CNN as example)

Initial State
Encoder
Network
Net2Deeper
Actor
Bi-LSTM
Figure 3: Net2Deeper actor, which uses a recurrent network
to sequentially determine where to insert the new layer and
corresponding parameters for the new layer based on the ﬁ-
nal hidden state of the encoder network given the input ar-
chitecture.

View Slide

Function-preserving Transformation
for DenseNet
16
• スキップ結合を持つネットワーク構造でも
function-preservingを実現する方法を考案
• DenseBlockの中にNet2Wider or Deeperを適用
Convolutional Networks
ng Liu∗
University
s.tsinghua.edu.cn
Laurens van der Maaten
Facebook AI Research
[email protected]
Q. Weinberger
ll University
cornell.edu
n
n
o
e
-
r
s
L
t
r
e
s
-
t
-
-
x0
x1
H1
x2
H2
H3
H4
x3
x4
Figure 1: A 5-layer dense block with a growth rate of k = 4.
Each layer takes all preceding feature-maps as input.
Convolution
Pooling
n lo
Convolution
Pooling
Pooling
in
Convolution
n ut
P i tion
o
n lo n lo
Figure 2: A deep DenseNet with three dense blocks. The layers between two adjacent blocks are referred to as transition layers and change
feature-map sizes via convolution and pooling.
ResNets can improve its performance provided the depth is
sufficient [41]. FractalNets also achieve competitive results
on several datasets using a wide network structure [17].
Instead of drawing representational power from ex-
tremely deep or wide architectures, DenseNets exploit the
potential of the network through feature reuse, yielding con-
densed models that are easy to train and highly parameter-
efficient. Concatenating feature-maps learned by different
layers increases variation in the input of subsequent layers
and improves efficiency. This constitutes a major difference
between DenseNets and ResNets. Compared to Inception
networks [35, 36], which also concatenate features from dif-
ferent layers, DenseNets are simpler and more efficient.
An advantage of ResNets is that the gradient can flow di-
rectly through the identity function from later layers to the
earlier layers. However, the identity function and the output
of Hℓ are combined by summation, which may impede the
information flow in the network.
Dense connectivity. To further improve the information
flow between layers we propose a different connectivity
pattern: we introduce direct connections from any layer
to all subsequent layers. Figure 1 illustrates the layout of
the resulting DenseNet schematically. Consequently, the
ℓth layer receives the feature-maps of all preceding layers,
x0, . . . , xℓ−1, as input:
密な結合を
持つブロック
DenseNetの構成図[Huang et al., 2017]より
ダウンサンプリング
前層までの出力を
全て結合して次の層に入力：
Ol = Fl ([O0, O1, . . . , Ol 1
])
AAAE3XicnVO/b9NAFH4pAUr40RQWJBaLtKigEl3KQIqEVAmE6IDapk1bqY4i27mkJo5t2ZfQEmVkQYi1AxNIDIiB/4CFhX+gQxeYEWORWBj47jktgpIgYct+v7/33bs7O/TcWAmxlxo5lj5+4uToqczpM2fPjWXHz6/GQTtyZNkJvCBat61Yeq4vy8pVnlwPI2m1bE+u2c07Or7WkVHsBv6K2g5lpWU1fLfuOpaCq5qdX6h2vZ5x27jH0vRkXU1B2JHlNA0ERW9aiwKEWQtUzJZ3vYDcKEkyI7exqa5WszmRn5ktihuzxlGlkBf85Kj/LAbjqfdkUo0CcqhNLZLkk4LukUUx3g0qkKAQvgp16SE8FkWwXM6R1KMM6tvwS2RZ8Dbxb8Da6Ht92Bo3ZgQHnTx8ESoNmhS74o3YFx/FW/FF/BiI1WUMzWcb0k5qZVgde3px+fs/q1qQijZ/VQ3lrKhORebqgnvIHr0KJ6nvPN7ZX75VmuxeEa/EV/B/KfbEB6zA73xzXi/J0gtGv4uaZIoRtAd9BgvAk/BoS8/gGrqYnNMAS92vx6y1bdL0oe9/ES3aOoKY+AbPTCFeB5Y+B/HQTBtoSdyH/Yj3tsUdfJwozaPE3c3DHbDxdtnLs8Sn12JhVQZ6utA9VA5DXPkDMYLWZe8BouJp+GAcolrzHobnstbjc175C9d5xEzIHOImd8nwfZHga1KH9ySZvM5I6m3YW6zbfNprh5gGTXDeRB8pg/t6cCmNwcrqTL4g8oUlkZsr9m/uKF2iyzQFrJs0R/dpkcpg8o526RN9TlfTT9LP0s+T1JFUv+YC/fakd34CfSoQkg==
AAAE0nicnVO/b9NAFH4pAUooNGVCYrFoiwoq0ZkFhISEBEJ0QG3Tpq1UR5HtXFITx7bsS2ixMrIgxMrABBIDYuA/YGHhH+jQBWbEWCQWBr57DkE0NJWwZb/f3/vu3Z0T+V6ihNjLjR3LHz9xcvxU4fTEmbOTxamJtSTsxK6suKEfxhuOnUjfC2RFecqXG1Es7bbjy3WndUfH17syTrwwWFU7kay27WbgNTzXVnDViguLtdTvGbeMeywtXzbUHIQT227LQFD05rUwIax6qBK2/KsmcuMsyYq95pa6XCtOi5LgxxhWzL4yTf1nKZzKfSCL6hSSSx1qk6SAFHSfbErwbpJJgiL4qpTSQ3hsimF5nCOpRwXUd+CXyLLhbeHfhLXZ9wawNW7CCC46+fhiVBo0K3bFW7EvPol34qv4eShWyhiazw6kk9XKqDb59PzKjyOr2pCKtv5UjeSsqEE3mKsH7hF79CrcrL77+MX+ys3ybHpJvBbfwP+V2BMfsYKg+919syzLLxn9LmqyKcbQHvQZLAJPwqMtPYMr6GJxThMsdb8es9a2RfMD3/8i2rQ9hJj5Dp+ZQrwBLH0OkpGZDtCyeAD7Ee9tmzsEOFGaR5m7W4MdcPCm7OVZ4tNrsbEqAz096D4qRyGuHkCMoaXs/Y2oeBoBGEeo1rxH4Xms9ficV//BdQExC3IacYu7FPi+SPC1qMt7kk1eZ2T1Duxt1h0+7fUBpkEznDfTRyrgvpoHb+ewsnatZIqSuSxonC7QRZoDxHW6TfdpiSog8J526TN9ydfyT/LPsps9lutf8XP015N//gtCcg74
AAAE0nicnVO/b9NAFH4pAUooNGVCYrFIiwoq0aUMtEhISCBEB9Q2bdpKdRTZziU1cWzLvoQWKyMLQqwMTCAxIAb+AxYW/oEOXWBGjEViYeC757QISoKELfv9/t537+7s0HNjJcReZuRY9viJk6OncqfHzpwdz0+MrcVBJ3JkxQm8INqwrVh6ri8rylWe3AgjabVtT67brds6vt6VUewG/qraCWW1bTV9t+E6loKrll9YrCVez7hp3GVperKhpiHsyHJaBoKiN6NFCcKsBypmy7taQm6UJpmR29xSl2v5gijOzs+Ja/PGUaVUFPwUqP8sBROZ92RSnQJyqENtkuSTgu6RRTHeTSqRoBC+KiX0AB6LIlgu50jqUQ71Hfglsix4W/g3YW32vT5sjRszgoNOHr4IlQZNiV3xRuyLj+Kt+CJ+DMRKGEPz2YG001oZ1safnF/5/s+qNqSirV9VQzkratAcc3XBPWSPXoWT1ncfPd9fuVGeSi6JV+Ir+L8Ue+IDVuB3vzmvl2X5BaPfQU06xQja/T6DReBJeLSlZ3AFXUzOaYKl7tdj1to2aebQ97+IFm0fQUx9g2emEG8AS5+DeGimDbQ07sN+yHvb5g4+TpTmUebu5uEO2HgT9vIs8em1WFiVgZ4udA+VwxBX/0CMoCXsPUBUPA0fjENUa97D8FzWenzOq3/huoCYCVlA3OQuOb4vEnxN6vKepJPXGWm9DXubdZtPe/0Q06BJzpvsI+VwXw8upTFYWZstlkSxtCxolC7QRZoGxHW6RfdoiSog8I526RN9ztayj7NP05s9kulf8XP025N99hO9LA9V
DenseBlockの特徴
O0, O1, . . . , Ol 1
AAAEsHicnVNNb9NAEJ0UAyV8NIULEhws0iKESrQJSLScKsEBDqht2rSR6iiynU1w69iW7YQWKxeO/AEOnEBCAnGCH8CFC3+AQ38C4lgkLhx4O3aDoCSVsGXvzNuZN29nd63AdaJYiL3cxDHt+ImTk6fyp8+cPTdVmD6/Hvm90JY123f9sG6ZkXQdT9ZiJ3ZlPQil2bVcuWFt31XzG30ZRo7vrcW7gWx0zY7ntB3bjAE1C5eXmokYzOkYyhiMlh9H7Lk3yoNmoShKlYV5cXNBP2yUS4KfImXPsj+d+0gGtcgnm3rUJUkexbBdMinCu0llEhQAa1BCW0BMCuE5HCNpQHnk94BLRJlAt/HvwNvMUA++4o2YwUYlF1+ITJ1mxRfxVuyLz+Kd+Cp+juRKmEPp2cVopbkyaE49u7j648isLsaYHv3OGqs5pjbNs1YH2gNG1CrsNL//5Pn+6p3qbHJVvBLfoP+l2BOfsAKv/91+vSKrL5j9HnLSLoawHmYKlsAngShP9eA6qhgc04FKVW/AqpVv0NwQ+19Gk3YOMabY6J7FmG+DS52DaGykBbZ03oP/mPe2yxU8nCilo8rVjeEOWHgTRrmX+NRaTKxKR00HtovMcYxrfzGGsBJGDxhj7oYHxQGyle5xfA5bAz7njX9ofYA5A2MR8wZXyfN9kdBrUJ/3JO28ikjzLfg7bFt82ltDTp1mOG4mY8rjvh5cSn20sV4plUWpvCKKi7eymztJl+gKXQPXbVqk+7RMNSh5Sm/oPX3QKlpda2pmGjqRy3Iu0B+PtvUL+DD/Fw==
Fl
AAAEl3icnVNNb9NAEJ0UAyV8tIULiEtEWoQQijYFiZYLlUBVOSCatGkr1VFku5uw1F+yndBi5Q9w4QiIE0gcEP+BCxf+AIf+BMSxSFw48HbsBkFJkFjL9szbmTdvZ3ft0FVxIsReYeyIcfTY8fETxZOnTp+ZmJw6uxYH3ciRDSdwg2jDtmLpKl82EpW4ciOMpOXZrly3t+/o+fWejGIV+KvJbiibntXxVVs5VgKosdhK3X5rsiwqs/Nz4vp86bBRrQgeZcrHcjBV+EAmbVFADnXJI0k+JbBdsijGs0lVEhQCa1JKj4BYFMFTHCOpT0Xkd4FLRFlAt/HtwNvMUR++5o2ZwUElF2+EzBLNiM/indgXn8R78UX8GMqVMofWs4u/neXKsDXx9PzK939mefgn9PBX1kjNCbVpjrUqaA8Z0atwsvzek+f7K7fqM+ll8UZ8hf7XYk98xAr83jfnbU3WXzH7XeRkXYxg3c8VPACfBKI93YOrqGJyTAcqdb0+q9a+SdcG2P8yWrRziDHDhvcswXwbXPocxCMjbbBl8z78x7y3HlfwcaK0jjpXNwc7YONJGeVe4tVrsbCqEmoq2C4yRzGu/sEYwUoZPWBMuBs+FIfI1rpH8Sm2+nzOm3/Reg9zJv5lzJtcpcj3RUKvST3ek6zzOiLLt+HvsG3zad8acJZomuOmc6Yi7uvBpSwNN9ZmK1VRqdZEeeFGfnPH6SJdoivgukkLtETL1IASRc/oBb00Lhi3jUVjKQsdK+Q55+i3YdR+Ap/m9vI=
AAAEjHicnVM9bxNBEB2HA4IJJKEC0Vg4QQgha00DogEJFIUCkThxEilnWXeXtbPkvnS3Ngkn/wEaSkBUIFEg/gMNDX+AIj8BUQaJhoK3c4cRMTESe7q7mbczb97O7rqxr1ItxH5p4ph1/MTJyVPl01Nnzk7PzE6tpVEv8WTTi/wo2XCdVPoqlE2ttC834kQ6gevLdXfnrplf78skVVG4qvdi2Qqcbqg6ynM0oOZCO/MH7ZmqqAkelVGjXhhVKsZSNFv6QDZtUUQe9SggSSFp2D45lOLZpDoJioG1KKNHQBxK4CmOkTSgMvJ7wCWiHKA7+HbhbRZoCN/wpszgoZKPN0FmhebFZ/FOHIhP4r34In4cyZUxh9Gzh7+b58q4Pf30/Mr3f2YF+Gva/p01VrOmDt1krQraY0bMKrw8v//k+cHKrcZ8dlm8EV+h/7XYFx+xgrD/zXu7LBuvmP0ecvIuJrAeFAoegk8CMZ7pwVVUsTmmC5Wm3oBVG9+ma0Psfxkd2h1hzLGje6Yx3wGXOQfp2EgXbPl8CP8x723AFUKcKKOjwdXt4Q64eDJGuZd4zVocrKqCmgq2j8xxjKuHGBNYGaO/GDV3I4TiGNlG9zg+xdaAz3nrL1rvY87Gv4p5m6uU+b5I6LWpz3uSd95E5Pku/F22XT7tW0POCs1x3FzBVMZ9rR++naPG2vVaXdTqy4Im6SJdoiuguEF3aJGWqAkBip7RC3ppXbBuWwv5zZ4oFVf8HP0xrMWfcuz1og==
AAAEjHicnVNNb9NAEJ0UAyUU2nICcbFIixBC0aYcaLmABKrKAdGmTVupjiLb3YSl/pLthBYrf4ALR0CcQOKA+A9cuPAHOPQnII5F4sKBt2M3CEqCxFq2Z97OvHk7u+tEnkpSIfZLY8eM4ydOjp8qn544c3ZyanpiPQm7sSsbbuiF8aZjJ9JTgWykKvXkZhRL23c8ueHs3NHzGz0ZJyoM1tK9SDZ9uxOotnLtFFBjsZV5/dZURVTnFubF9QXzqFGrCh4VKsZyOF36QBZtU0gudcknSQGlsD2yKcGzRTUSFAFrUkaPgNgUw1McI6lPZeR3gUtE2UB38O3A2yrQAL7mTZjBRSUPb4xMk2bFZ/FOHIhP4r34In4M5cqYQ+vZw9/Jc2XUmnx6fvX7P7N8/FN6+CtrpOaU2jTPWhW0R4zoVbh5fu/J84PVm/XZ7LJ4I75C/2uxLz5iBUHvm/t2RdZfMftd5ORdjGHdLxQ8AJ8Eoj3dg6uoYnFMByp1vT6r1r5F1wbY/zLatHuEMceG9yzFfBtc+hwkIyMdsOXzAfzHvLc+VwhworSOOle3Bjvg4MkY5V7i1WuxsSoTNRVsD5mjGNf+YIxhZYweMqbcjQCKI2Rr3aP4FFt9PufNv2i9hzkL/wrmLa5S5vsiodeiHu9J3nkdkec78HfZdvi0bw84TZrhuJmCqYz7engpzeHG+ly1Jqq1FUHjdJEu0RVQ3KDbtETL1IAARc/oBb00Lhi3jMX8Zo+Viit+jn4bxtJP7ab1/w==

View Slide

for DenseNet
17
• 層目にNet2WiderNetが行われたときの
層目( )の入力は次の関数で決定：
l
AAAEk3icnVNNb9NAEJ0UAyV8tAUhIXGJSIsQQtGmPbQFDhX0AIeKJm3aSnVU2e4mLPWXbCe0WPkDcAVx4AQSB8R/4MKFP8ChPwFxLBIXDrwdu0FQEiTWsj3zdubN29ldO3RVnAixXxg5Zhw/cXL0VPH0mbPnxsYnzq/FQSdyZMMJ3CDasK1YusqXjUQlrtwII2l5tivX7Z27en69K6NYBf5qshfKpme1fdVSjpUAqrlb42VRmZ6fEzPzpaNGtSJ4lCkfy8FE4QOZtE0BOdQhjyT5lMB2yaIYzyZVSVAIrEkpPQJiUQRPcYykHhWR3wEuEWUB3cG3DW8zR334mjdmBgeVXLwRMks0JT6Ld+JAfBLvxRfxYyBXyhxazx7+dpYrw62xp5dWvv8zy8M/oYe/soZqTqhFc6xVQXvIiF6Fk+V3n7w8WLlZn0qvijfiK/S/FvviI1bgd785b2uy/orZF5GTdTGCtZQreAA+CUR7ugfXUcXkmDZU6no9Vq19k270sf9ltGj3CGOGDe5ZgvkWuPQ5iIdG2mDL5n34j3lvPa7g40RpHXWubvZ3wMaTMsq9xKvXYmFVJdRUsF1kDmNc/YMxgpUyesiYcDd8KA6RrXUP41Ns9ficN/+i9T7mTPzLmDe5SpHvi4Rek7q8J1nndUSWb8PfZdvm077d5yzRJMdN5kxF3NfDS1kabKxNV6qiUq2J8sLt/OaO0mW6QtfANUsLdI+WqQElkp7Rc3phXDRuGXeMxSx0pJDnXKDfhrH0E0Az9TU=
m
AAAEk3icnVNNb9NAEJ0UAyV8tAUhIXGJSIsQQtGmPbQFCVXQAxwqmrRpK9VRZbubYOov2ZvQYuUPwBXEgRNIHBD/gQsX/gCH/gTEsUhcOPB27AZBSZBYy/bM25k3b2d37chzEyXEfmHkmHH8xMnRU8XTZ86eGxufOL+WhJ3YkQ0n9MJ4w7YS6bmBbChXeXIjiqXl255ct3fu6vn1rowTNwxW1V4km77VDtyW61gKUM3fGi+LyvT8nJiZLx01qhXBo0z5WA4nCh/IpG0KyaEO+SQpIAXbI4sSPJtUJUERsCal9AiIRTE8l2Mk9aiI/A5wiSgL6A6+bXibORrA17wJMzio5OGNkVmiKfFZvBMH4pN4L76IHwO5UubQevbwt7NcGW2NPb208v2fWT7+ih7+yhqqWVGL5lirC+0RI3oVTpbfffLyYOVmfSq9Kt6Ir9D/WuyLj1hB0P3mvK3J+itmX0RO1sUY1lKu4AH4JBDt6R5cRxWTY9pQqev1WLX2TbrRx/6X0aLdI4wZNrhnCvMtcOlzkAyNtMGWzQfwH/Pe+lwhwInSOupc3ezvgI0nZZR7iVevxcKqSqjpwvaQOYxx9Q/GGFbK6CGj4m4EUBwhW+sexuey1eNz3vyL1vuYM/EvY97kKkW+LxJ6TerynmSd1xFZvg1/l22bT/t2n7NEkxw3mTMVcV8PL2VpsLE2XamKSrUmygu385s7SpfpCl0D1ywt0D1apgaUSHpGz+mFcdG4ZdwxFrPQkUKec4F+G8bST0T89Tg=
AAAEiHicnVM9bxNBEB0HA8EEkiAhIdFYOEEIIWuPJhE0CFJAgYidOImUs6K7y9osuS/drU3CyX8AWhAFFUgUiP9AQ8MfoMhPQJRBoqHg7dxhREyMxJ7ububtzJu3s7tu7KtUC7FfmjhWPn7i5OSpyumpM2enZ2an1tKol3iy5UV+lGy4Tip9FcqWVtqXG3EincD15bq7c8fMr/dlkqooXNV7sWwHTjdUHeU5GlAj2JqpibrgUR01rMKoUTGWo9nSB7JpmyLyqEcBSQpJw/bJoRTPJlkkKAbWpoweAXEogac4RtKAKsjvAZeIcoDu4NuFt1mgIXzDmzKDh0o+3gSZVZoXn8U7cSA+iffii/hxJFfGHEbPHv5univjremnF1a+/zMrwF/Tw99ZYzVr6tAia1XQHjNiVuHl+f0nLw9WbjTns8vijfgK/a/FvviIFYT9b97bhmy+YvYl5ORdTGDdLxQ8AJ8EYjzTg6uoYnNMFypNvQGrNr5N14bY/zI6tDvCmGNH90xjvgMucw7SsZEu2PL5EP5j3tuAK4Q4UUZHk6vbwx1w8WSMci/xmrU4WFUVNRVsH5njGFcPMSawMkZ/MWruRgjFMbKN7nF8iq0Bn/P2X7Tew5yNfw3zNlep8H2R0GtTn/ck77yJyPNd+Ltsu3zat4ecVZrjuLmCqYL7ah2+naPG2vW6JepWQ9AkXaRLdAUUC3SL7tIytSBA0jN6Ti/K58s3y7fzmz1RKq74OfpjlJd+Aie08+I=
AAAEiHicnVNNb9NAEJ0UAyUU2iIhIXGxSIsQQtGmHGjhgqAHOCCatGkr1VFlu5uw1F+yndBi5Q/AFcSBE0gcEP+BCxf+AIf+BMSxSFw48HbsBkFJkLqW7Zm3M2/ezu46kaeSVIi90tgx4/iJk+OnyqcnzpydnJqeWE3CbuzKpht6Ybzu2In0VCCbqUo9uR7F0vYdT64523f1/FpPxokKg5V0N5It3+4Eqq1cOwVU9zenKqI6tzAvri+Yh41aVfCoUDGWwunSR7Joi0JyqUs+SQoohe2RTQmeDaqRoAhYizJ6DMSmGJ7iGEl9KiO/C1wiyga6jW8H3kaBBvA1b8IMLip5eGNkmjQrvoj3Yl98Fh/EV/FzKFfGHFrPLv5OniujzclnF5Z//DfLxz+lR7+zRmpOqU3zrFVBe8SIXoWb5/eevtpfvtmYzS6Lt+Ib9L8Re+ITVhD0vrvv6rLxmtkXkZN3MYb1oFDwEHwSiPZ0D66iisUxHajU9fqsWvsWXRtgR2W0aecQY44N71mK+Ta49DlIRkY6YMvnA/hPeG99rhDgRGkdDa5uDXbAwZMxyr3Eq9diY1UmairYHjJHMa78xRjDyhg9YEy5GwEUR8jWukfxKbb6fM5b/9B6H3MW/hXMW1ylzPdFQq9FPd6TvPM6Is934O+w7fBp3xpwmjTDcTMFUxn39eBSmsON1blqTVRrdUHjdJEu0RVQ3KDbdI+WqAkBkp7TC3ppnDduGXfymz1WKq74OfpjGIu/AKJu9D8=
ˆ
Gm
AAAEnXicnVNNb9NAEJ0UAyV8tIULEgci0iKEULQpB1oOqBIgQOKjTZumUhxFtrtJTf0l2wktlv8Ad8QBCQQSB8R/4MKFP8ChPwFxLBIXDrwdu0HQJkisZXvm7cybt7O7ZuDYUSzETmHskHb4yNHxY8XjJ06empicOr0a+b3QknXLd/xwzTQi6dierMd27Mi1IJSGazqyYW7eVPONvgwj2/dW4u1Atlyj69kd2zJiQE19w4iTO2k7cdP2ZFlUZufnxNX50n6jWhE8ypSPRX+q8JF0WiefLOqRS5I8imE7ZFCEp0lVEhQAa1FCj4EYFMKzOUZSSkXk94BLRBlAN/HtwmvmqAdf8UbMYKGSgzdEZolmxBfxXuyKz+KD+Cp+DuVKmEPp2cbfzHJl0J54dnb5xz+zXPxj2vidNVJzTB2aY602tAeMqFVYWX7/6Yvd5eu1meSieCu+Qf8bsSM+YQVe/7v1bknWXjL7LeRkXQxhPcgVPAKfBKI81YPLqKJzTBcqVb2UVStfpysD7H8ZDdrax5hhw3sWY74DLnUOopGRJtiyeQ/+E95blyt4OFFKR42r64MdMPEkjHIv8aq1GFhVCTVt2A4yRzGu/MUYwkoY3WOMuRseFAfIVrpH8dlspXzOWwdovYc5Hf8y5nWuUuT7IqFXpz7vSdZ5FZHlm/C32Db5tK8POEs0zXHTOVMR93XvUpaGG6uzlaqoVJdEeeFGfnPH6RxdoEvgukYLdJcWqc6re06v6LV2Xrut3dceZqFjhTznDP0xtMYvaNf5yw==
AAAEknicnVNNb9NAEJ2UACUU2nJC4kBEWoQQitZcWnFCAgRIfLRp01aKo8h2N6mpv2RvQovlP8AdcUACgcQB8R+4cOEPcOhPQByLxIUDb8cmiIYGibVsz7ydefN2dteOPDdRQuyVJo6Ujx47PnmicnLq1OnpmdmptSTsx45sOqEXxhu2lUjPDWRTucqTG1EsLd/25Lq9fUPPrw9knLhhsKp2I9n2rV7gdl3HUoBa5pal0ttZJ/WzzkxN1AWP6qhhFEaNirEUzpY+kEmbFJJDffJJUkAKtkcWJXhaZJCgCFibUnoExKIYnssxkjKqIL8PXCLKArqNbw9eq0AD+Jo3YQYHlTy8MTKrNC8+i3diX3wS78UX8eNQrpQ5tJ5d/O08V0ad6adnV77/M8vHX9HW76yxmhV1aZG1utAeMaJX4eT5gyfP91euNebTi+KN+Ar9r8We+IgVBINvzttl2XjB7DeRk3cxhnW/UPAQfBKI9nQPLqOKyTE9qNT1MlatfZOuDLH/ZbRoZ4Qxxw7vmcJ8F1z6HCRjI22w5fMB/Me8tz5XCHCitI4GVzeHO2DjSRnlXuLVa7GwqipqurA9ZI5jXD3AGMNKGf3FqLgbARRHyNa6x/G5bGV8ztt/0XoXcyb+NcybXKXC90VCr0kD3pO88zoiz7fh77Bt82nfHHJWaY7j5gqmCu6rcfB2jhprV+uGqBvLgibpHF2gS6BYoOt0h5aoyYt6Ri/pVfl8+Vb5Xn6zJ0rFFT9Df4zyg58bCvhr
AAAEknicnVNNb9NAEJ0UAyUU2nJC4oBFWoQQijblQMsJCRAg8dGmTVspjiLb3aSm/pK9CS1W/gB3xAEJBBIHxH/gwoU/wKE/AXEsEhcOvB2nQVCSSqxle+btzJu3s7tO7HupEmK3MHbEOHrs+PiJ4smJU6cnp6YnVtOok7iy5kZ+lKw7dip9L5Q15SlfrseJtAPHl2vO1k09v9aVSepF4YraiWUjsNuh1/JcWwGqW5u2yu70mlnQa06VRHluYV5cXTAPGpWy4FGi/liMpgsfyaINisilDgUkKSQF2yebUjx1qpCgGFiDMnoMxKYEnscxknpURH4HuESUDXQL3za8eh8N4WvelBlcVPLxJsg0aVZ8Ee/FnvgsPoiv4udQrow5tJ4d/J08V8bNyWdnl38cmhXgr2jzd9ZIzYpaNM9aPWiPGdGrcPP87tMXe8vXq7PZRfFWfIP+N2JXfMIKwu53992SrL5k9lvIybuYwHrQV/AIfBKI9nQPLqOKxTFtqNT1eqxa+xZdGWD/y2jT9gHGHBveM4X5Frj0OUhHRjpgy+dD+E94bwOuEOJEaR1Vrm4NdsDBkzHKvcSr12JjVSZqerB9ZI5iXPmLMYGVMbrPqLgbIRTHyNa6R/F5bPX4nDf+ofUe5iz8S5i3uEqR74uEXou6vCd553VEnu/A32bb4dO+MeA0aYbjZvpMRdzX/UtpDjdW58oVUa4sCRqnc3SBLoHiGt2gu7RINV7Uc3pFr43zxm3jfn6zxwr9K36G/hjGw1+VxPjI
l < m
AAAEl3icnVNNb9NAEJ0UAyV8tIULiItFWoQQijblQIqQqARC5YBo0qatVEeV7W7CUn/JdkJLlD/AhSMgTiBxQPwHLlz4Axz6ExDHInHhwNuxGwQlqcRatmfezrx5O7vrRJ5KUiF2C2NHjKPHjo+fKJ48dfrMxOTU2ZUk7MSubLihF8Zrjp1ITwWykarUk2tRLG3f8eSqs3VHz692ZZyoMFhOdyLZ9O12oFrKtVNADc+8ZfobkyVRnp2riutz5kGjUhY8SpSPxXCq8JEs2qSQXOqQT5ICSmF7ZFOCZ50qJCgC1qQePQZiUwxPcYykPhWR3wEuEWUD3cK3DW89RwP4mjdhBheVPLwxMk2aEV/Ee7EnPosP4qv4OZSrxxxazw7+TpYro42JZ+eXfhya5eOf0qPfWSM1p9SiKmtV0B4xolfhZvndpy/2lm7WZ3qXxVvxDfrfiF3xCSsIut/ddzVZf83sd5GTdTGG9SBX8BB8Eoj2dA+uoorFMW2o1PX6rFr7Fl0bYP/LaNP2AcYMG96zFPMtcOlzkIyMdMCWzQfwn/De+lwhwInSOupc3RrsgIOnxyj3Eq9ei41VmaipYHvIHMW4/BdjDKvH6D5jyt0IoDhCttY9ik+x1edz3vyH1vuYs/AvYd7iKkW+LxJ6LerynmSd1xFZvgN/m22HT/vmgNOkaY6bzpmKuK/7l9IcbqzMliuiXKmJ0nw1v7njdJEu0RVw3aB5WqBFakCJouf0kl4ZF4zbxj1jIQsdK+Q55+iPYdR+Aa7y9kI=
AAAEjHicnVPPa9RAFH5bo9a12taT4iW4rYjIMvFiKYKCIvUgtttuW2iWkqSz27H5RZJdW8P+A148qnhS8CD+D168+A946J8gHit48eA3L3HFrl3BCUne++a9733zZsaNfZVmQuxXxo4Zx0+cHD9VPT1x5uzk1PTEahp1E082vciPknXXSaWvQtnMVObL9TiRTuD6cs3duaPn13oySVUUrmR7sWwFTidUbeU5GaCmb940g82pmqgLHuawYZVGjcqxGE1XPpBNWxSRR10KSFJIGWyfHErxbJBFgmJgLcrpERCHEniKYyT1qYr8LnCJKAfoDr4deBslGsLXvCkzeKjk402QadKs+CzeiQPxSbwXX8SPI7ly5tB69vB3i1wZb04+Pb/8/Z9ZAf4Zbf/OGqk5ozbNsVYF7TEjehVekd978vxgeb4xm18Wb8RX6H8t9sVHrCDsffPeLsnGK2a/i5yiiwmsB6WCh+CTQLSne3AVVWyO6UClrtdn1dq36doA+19Gh3aHGAvs6J5lmG+DS5+DdGSkC7ZiPoT/mPc24AohTpTW0eDq9mAHXDw5o9xLvHotDlZloqaC7SNzFOPKIcYEVs7oL8aMuxFCcYxsrXsUn2Krz+e89Ret9zFn41/DvM1VqnxfJPTa1OM9KTqvI4p8F/4u2y6f9q0Bp0kzHDdTMlVxX63Dt3PYWL1et0TdWhI0ThfpEl0BxQ26TQu0SE0IUPSMXtBL44Jxy7hX3OyxSnnFz9Efw1j4CYiA9O4=
AAAEjHicnVNNb9NAEJ0UAyUU2nICcbFIixBC0aY90CIkkECoHBBt2rSV6qiy3U3Y1l+yndBi5Q9w4QiIE0gcEP+BCxf+AIf+BMSxSFw48HbsBkFJkFjL9szbmTdvZ3edyFNJKsR+aeSYcfzEydFT5dNjZ86OT0yOrSZhJ3Zlww29MF537ER6KpCNVKWeXI9iafuOJ9ecnTt6fq0r40SFwUq6F8mmb7cD1VKunQJqeOZN09+cqIjqzPycmJ03jxq1quBRoWIshpOlD2TRFoXkUod8khRQCtsjmxI8G1QjQRGwJmW0DcSmGJ7iGEk9KiO/A1wiyga6g28b3kaBBvA1b8IMLip5eGNkmjQtPot34kB8Eu/FF/FjIFfGHFrPHv5OniujzfGn55e//zPLxz+lR7+yhmpOqUVzrFVBe8SIXoWb53efPD9YvlGfzi6LN+Ir9L8W++IjVhB0v7lvl2T9FbPfRU7exRjWg0LBQ/BJINrTPbiKKhbHtKFS1+uxau1bdK2P/S+jTbtHGHNscM9SzLfApc9BMjTSAVs+H8B/zHvrc4UAJ0rrqHN1q78DDp6MUe4lXr0WG6syUVPB9pA5jHHlD8YYVsboIWPK3QigOEK21j2MT7HV43Pe/IvW+5iz8K9g3uIqZb4vEnot6vKe5J3XEXm+A3+XbYdP+1af06QpjpsqmMq4r4eX0hxsrM5Ua6JaWxI0ShfpEl0BxXW6TQu0SA0IUPSMXtBL44Jxy7iX3+yRUnHFz9Fvw1j4CQNJ9Us=
：層目の入力チャネル数
l
f
0:
l
o
=
l
1
X
v
=0
fv
o
AAAEvnicnVM9bxNBEB0HA8F8xIEGieaEE4QisNahwImEZAkKKBCJEyeRco51d1mbI/elu7NxON0foKKjoAKJAtFTAhINf4AiPwFRBomGgrdzjhEEOxJ7uruZtzNv3s7umoFjR7EQe7mJY/njJ05OniqcPnP23FRx+vxa5HdDSzYs3/HDDdOIpGN7shHbsSM3glAarunIdXPntppf78kwsn1vNd4NZNM1Op7dti0jBtQqzrW3ErHopK3ET7Vbmh513a3EuV4B0IMvUg0BPZ5uFUuiPL9QFTcWtMNGpSx4lGgwlvzp3HvSaZt8sqhLLknyKIbtkEERnk2qkKAAWJMSegTEoBCezTGSUiogvwtcIsoAuoNvB97mAPXgK96IGSxUcvCGyNRoVnwRb8S++Czeiq/i50iuhDmUnl38zSxXBq2ppxdXfhyZ5eIf08PfWWM1x9SmKmu1oT1gRK3CyvJ7T57vryzWZ5Mr4pX4Bv0vxZ74hBV4ve/W62VZf8Hsd5CTdTGEdX+g4AH4JBDlqR7MoYrOMR2oVPVSVq18na4Nsf9lNKh/iDHDRvcsxnwbXOocRGMjTbBl8x78x7y3LlfwcKKUjjpX14c7YOJJGOVe4lVrMbAqDTVt2A4yxzGu/sUYwkoYPWCMuRseFAfIVrrH8dlspXzOm//Qeg9zOv4lzOtcpcD3RUKvTj3ek6zzKiLLN+H32Tb5tG8POTWa4biZAVMB9/XgUmqjjbX5ckWUK8uiVKsObu4kXaLLdBVcN6lGd2mJGlDyjN7RB/qYr+XbeTfvZ6ETuUHOBfpj5Pu/AIdQBdc=
AAAEs3icnVM9b9NQFL0pAUooNGVCYrFIi1AF0TMLqBJSJRhgQLRp01aq08h2X4Kpv2Q7psXyH2BiY2ACiQGxMwISC3+AoT8BMRaJhYHzrk0QDQ0Sz7J977n3nnvelxW6TpwIsV+ZOFY9fuLk5Kna6akzZ6frM1NrcTCIbNm2AzeINiwzlq7jy3biJK7cCCNpepYr162dWyq+nsoodgJ/NdkLZccz+77Tc2wzAdStz/e2MrHg5t0syLWbmhEPvK3MvaoDSOGLXENCyuFuvSGagoc2auil0aByLAUzlXdk0DYFZNOAPJLkUwLbJZNiPJukk6AQWIcyegjEpAiewzmScqqhfgBcIssEuoNvH95mifrwFW/MDDY6uXgjVGo0Jz6L1+JAfBJvxBfx40iujDmUnj38raJWht3pJ+dXvv+zysM/oQe/q8ZqTqhHN1irA+0hI2oWdlGfPn52sLLQmssuiZfiK/S/EPviI2bgp9/sV8uy9ZzZb6OmWMUI1r1SwX3wSSDKU2swjy4G5/ShUvXLWbXyDboyxP6X0aTdEcYCO3rNEsR74FLnIB6baYGtiPvwH/HeetzBx4lSOlrc3RjugIUnY5TXEq+ai4lZaejpwHZROY5x9RBjBCtj9BdjwqvhQ3GIaqV7HJ/DVs7nvPMXrXcRM/BvIG5wlxrfFwm9BqW8J8XKq4yi3oK/y7bFp317yKnRLOfNlkw13Ff98O0cNdauNXXR1JcFTdIFukiXQXGdFukOLVEbAp7SW3pPH6qL1V7VK272RKW84ufoj1ENfgK9DwRc
AAAEs3icnVM9b9NQFL0pAUooNGVCYrFIi1AF0Us79ENCqgQDDIg2bdpKdRrZ7ksw9Zdsx7RY/gNMbAxMIDEgdkZAYuEPMPQnIMYisTBw3nUaBCWpxLNs33vuveee92UGjh3FQhwURk4VT585O3qudH7swsXx8sTYeuR3Q0s2LN/xw03TiKRje7IR27EjN4NQGq7pyA1z97aKbyQyjGzfW4v3A9l0jY5nt23LiAG1ytPt7VQsOlkr9TPtlqZHXXc7dW7WACTwRaYhIeFwq1wR1ZmFeTG7oB03alXBo0K9sexPFN6TTjvkk0VdckmSRzFshwyK8GxRjQQFwJqU0iMgBoXwbM6RlFEJ9V3gElkG0F18O/C2eqgHX/FGzGChk4M3RKVGU+KLeCMOxWfxVnwVPwdypcyh9Ozjb+a1MmiNP728+uPEKhf/mB7+rhqqOaY2zbNWG9oDRtQsrLw+efL8cHWxPpVeE6/EN+h/KQ7EJ8zAS75br1dk/QWz30FNvoohrPs9BQ/AJ4EoT63BNLronNOBStUvY9XK1+lGH/tfRoP2jjHm2OA1ixFvg0udg2hopgm2PO7Bf8x763IHDydK6ahzd72/AyaelFFeS7xqLgZmpaGnDdtB5TDGtb8YQ1gpo0eMMa+GB8UBqpXuYXw2Wxmf8+Y/tN5DTMe/grjOXUp8XyT06pTwnuQrrzLyehP+Htsmn/adPqdGk5w32WMq4b4eXUptsLE+U62Jam1F0Chdoat0HRRztER3aZkaEPCM3tEH+lhcKraLbn6zRwq9K36J/hhF/xc32AS5
層目までの出力はそのまま
l 1
AAAEl3icnVNNb9NAEJ0UAyV8tIULiItFWoQQRJtyIOVCJRAqB0STNm2lOqpsdxOWOrZlO6HFyh/gwhEQJ5A4IP4DFy78AQ79CYhjkbhw4O3YDYKSVGIt2zNvZ968nd11Qk/FiRC7hbEjxtFjx8dPFE+eOn1mYnLq7EocdCNXNtzAC6I1x46lp3zZSFTiybUwknbH8eSqs3VHz6/2ZBSrwF9OdkLZ7NhtX7WUayeAGp553axsTJZEeXauKm7MmQeNSlnwKFE+FoOpwkeyaJMCcqlLHZLkUwLbI5tiPOtUIUEhsCal9BiITRE8xTGS+lREfhe4RJQNdAvfNrz1HPXha96YGVxU8vBGyDRpRnwR78We+Cw+iK/i51CulDm0nh38nSxXhhsTz84v/Tg0q4N/Qo9+Z43UnFCLqqxVQXvIiF6Fm+X3nr7YW7pVn0kvi7fiG/S/EbviE1bg976772qy/prZ7yIn62IE60Gu4CH4JBDt6R5cRRWLY9pQqev1WbX2Lbo2wP6X0abtA4wZNrxnCeZb4NLnIB4Z6YAtm/fhP+G97XAFHydK66hzdWuwAw6elFHuJV69FhurMlFTwfaQOYpx+S/GCFbK6D5jwt3woThEttY9ik+x1edz3vyH1vuYs/AvYd7iKkW+LxJ6LerxnmSd1xFZvgN/m22HT/vmgNOkaY6bzpmKuK/7l9IcbqzMliuiXKmJ0nw1v7njdJEu0RVw3aR5WqBFakCJouf0kl4ZF4zbxj1jIQsdK+Q55+iPYdR+AXUA9fc=
AAAEjHicnVM9b9RAEJ0LBsIRSEIForG4BCFETmuaRDQggVAoEMkll0SKT5Ht7B1L/CXbdyRY9wdoKAFRgUSB+A80NPwBivwERBkkGgrejs0hcuSQWMv2zNuZN29nd93YV2kmxH5l7Jhx/MTJ8VPV0xNnzk5OTU+spVE38WTTi/wo2XCdVPoqlM1MZb7ciBPpBK4v192d23p+vSeTVEXharYXy1bgdELVVp6TAWr65pxpbU3VRF3wMIcNqzRqVI6laLrygWzapog86lJAkkLKYPvkUIpnkywSFANrUU6PgDiUwFMcI6lPVeR3gUtEOUB38O3A2yzREL7mTZnBQyUfb4JMk2bFZ/FOHIhP4r34In4cyZUzh9azh79b5Mp4a/Lp+ZXv/8wK8M/o4e+skZozatMCa1XQHjOiV+EV+b0nzw9WbjRm88vijfgK/a/FvviIFYS9b97bZdl4xex3kFN0MYF1v1TwAHwSiPZ0D66iis0xHajU9fqsWvs2XRtg/8vo0O4QY4Ed3bMM821w6XOQjox0wVbMh/Af894GXCHEidI6GlzdHuyAiydnlHuJV6/FwapM1FSwfWSOYlw9xJjAyhn9xZhxN0IojpGtdY/iU2z1+Zy3/qL1HuZs/GuYt7lKle+LhF6berwnRed1RJHvwt9l2+XTvj3gNGmG42ZKpiruq3X4dg4ba9frlqhby4LG6SJdoiugmKdbtEhL1IQARc/oBb00Lhg3jbvFzR6rlFf8HP0xjMWfUcf0ow==
AAAEjHicnVNNb9NAEJ2UACUU2nICcbFIixCCaN0e2nIBCYTKAdGmTVupjirb3YSl/pLthBYrf4ALR0CcQOKA+A9cuPAHOPQnII5F4sKBt2M3CEqCxFq2Z97OvHk7u+tEnkpSIfZLI8fKx0+cHD1VOT125uz4xOTYWhJ2Ylc23NAL4w3HTqSnAtlIVerJjSiWtu94ct3Zua3n17syTlQYrKZ7kWz6djtQLeXaKaCGZ1w3zK2JqqjNLMyL2QXjqGHWBI8qFWMpnCx9IIu2KSSXOuSTpIBS2B7ZlODZJJMERcCalNEjIDbF8BTHSOpRBfkd4BJRNtAdfNvwNgs0gK95E2ZwUcnDGyPToGnxWbwTB+KTeC++iB8DuTLm0Hr28HfyXBltjT89v/L9n1k+/ik9/JU1VHNKLZpnrQraI0b0Ktw8v/vk+cHKjfp0dlm8EV+h/7XYFx+xgqD7zX27LOuvmP0OcvIuxrDuFwoegE8C0Z7uwVVUsTimDZW6Xo9Va9+ia33sfxlt2j3CmGODe5ZivgUufQ6SoZEO2PL5AP5j3lufKwQ4UVpHnatb/R1w8GSMci/x6rXYWJWBmgq2h8xhjKt/MMawMkYPGVPuRgDFEbK17mF8iq0en/PmX7Tew5yFfxXzFlep8H2R0GtRl/ck77yOyPMd+LtsO3zat/ucBk1x3FTBVMF9PbyUxmBjbaZmipq5LGiULtIlugKKObpFi7REDQhQ9Ixe0MvyhfLN8t38Zo+Uiit+jn4b5cWfzIH1AA==
層目の出力は
関数で決定
l
Gl
層目以降の出力もそのまま
l + 1
AAAEl3icnVNNb9NAEJ0UAyV8tIULiItFWoQARZtyIOVCJRAqB0STNm2lOqpsdxOWOrZlO6HFyh/gwhEQJ5A4IP4DFy78AQ79CYhjkbhw4O3YDYKSVGIt2zNvZ968nd11Qk/FiRC7hbEjxtFjx8dPFE+eOn1mYnLq7EocdCNXNtzAC6I1x46lp3zZSFTiybUwknbH8eSqs3VHz6/2ZBSrwF9OdkLZ7NhtX7WUayeAGp55zaxsTJZEeXauKm7MmQeNSlnwKFE+FoOpwkeyaJMCcqlLHZLkUwLbI5tiPOtUIUEhsCal9BiITRE8xTGS+lREfhe4RJQNdAvfNrz1HPXha96YGVxU8vBGyDRpRnwR78We+Cw+iK/i51CulDm0nh38nSxXhhsTz84v/Tg0q4N/Qo9+Z43UnFCLqqxVQXvIiF6Fm+X3nr7YW7pVn0kvi7fiG/S/EbviE1bg976772qy/prZ7yIn62IE60Gu4CH4JBDt6R5cRRWLY9pQqev1WbX2Lbo+wP6X0abtA4wZNrxnCeZb4NLnIB4Z6YAtm/fhP+G97XAFHydK66hzdWuwAw6elFHuJV69FhurMlFTwfaQOYpx+S/GCFbK6D5jwt3woThEttY9ik+x1edz3vyH1vuYs/AvYd7iKkW+LxJ6LerxnmSd1xFZvgN/m22HT/vmgNOkaY6bzpmKuK/7l9IcbqzMliuiXKmJ0nw1v7njdJEu0RVw3aR5WqBFakCJouf0kl4ZF4zbxj1jIQsdK+Q55+iPYdR+AWye9fU=
AAAEjHicnVM9b9RAEJ0LBsIRSEIForG4BCGITmuaRDQggVAoEMkll0SKT5Ht7B1L/CXbdyRY9wdoKAFRgUSB+A80NPwBivwERBkkGgrejs0hcuSQWMv2zNuZN29nd93YV2kmxH5l7Jhx/MTJ8VPV0xNnzk5OTU+spVE38WTTi/wo2XCdVPoqlM1MZb7ciBPpBK4v192d23p+vSeTVEXharYXy1bgdELVVp6TAWr65jXT2pqqibrgYQ4bVmnUqBxL0XTlA9m0TRF51KWAJIWUwfbJoRTPJlkkKAbWopweAXEogac4RlKfqsjvApeIcoDu4NuBt1miIXzNmzKDh0o+3gSZJs2Kz+KdOBCfxHvxRfw4kitnDq1nD3+3yJXx1uTT8yvf/5kV4J/Rw99ZIzVn1KYF1qqgPWZEr8Ir8ntPnh+s3GjM5pfFG/EV+l+LffERKwh737y3y7LxitnvIKfoYgLrfqngAfgkEO3pHlxFFZtjOlCp6/VZtfZtmhtg/8vo0O4QY4Ed3bMM821w6XOQjox0wVbMh/Af894GXCHEidI6GlzdHuyAiydnlHuJV6/FwapM1FSwfWSOYlw9xJjAyhn9xZhxN0IojpGtdY/iU2z1+Zy3/qL1HuZs/GuYt7lKle+LhF6berwnRed1RJHvwt9l2+XTvj3gNGmG42ZKpiruq3X4dg4ba9frlqhby4LG6SJdoiugmKdbtEhL1IQARc/oBb00Lhg3jbvFzR6rlFf8HP0xjMWfSXv0oQ==
AAAEjHicnVNNb9NAEJ2UACUU2nICcbFIixCgaN0e2nIBCYTKAdGmTVupjirb3YSl/pLthBYrf4ALR0CcQOKA+A9cuPAHOPQnII5F4sKBt2M3CEqCxFq2Z97OvHk7u+tEnkpSIfZLI8fKx0+cHD1VOT125uz4xOTYWhJ2Ylc23NAL4w3HTqSnAtlIVerJjSiWtu94ct3Zua3n17syTlQYrKZ7kWz6djtQLeXaKaCGZ1wzzK2JqqjNLMyL2QXjqGHWBI8qFWMpnCx9IIu2KSSXOuSTpIBS2B7ZlODZJJMERcCalNEjIDbF8BTHSOpRBfkd4BJRNtAdfNvwNgs0gK95E2ZwUcnDGyPToGnxWbwTB+KTeC++iB8DuTLm0Hr28HfyXBltjT89v/L9n1k+/ik9/JU1VHNKLZpnrQraI0b0Ktw8v/vk+cHKjfp0dlm8EV+h/7XYFx+xgqD7zX27LOuvmP0OcvIuxrDuFwoegE8C0Z7uwVVUsTimDZW6Xo9Va9+i633sfxlt2j3CmGODe5ZivgUufQ6SoZEO2PL5AP5j3lufKwQ4UVpHnatb/R1w8GSMci/x6rXYWJWBmgq2h8xhjKt/MMawMkYPGVPuRgDFEbK17mF8iq0en/PmX7Tew5yFfxXzFlep8H2R0GtRl/ck77yOyPMd+LtsO3zat/ucBk1x3FTBVMF9PbyUxmBjbaZmipq5LGiULtIlugKKObpFi7REDQhQ9Ixe0MvyhfLN8t38Zo+Uiit+jn4b5cWfxDX0/g==
AAAEjHicnVM9b9RAEJ0LBsIRSEIForG4BCGITmuaRDQggVAoEMkll0SKT5Ht7B1L/CXbdyRY9wdoKAFRgUSB+A80NPwBivwERBkkGgrejs0hcuSQWMv2zNuZN29nd93YV2kmxH5l7Jhx/MTJ8VPV0xNnzk5OTU+spVE38WTTi/wo2XCdVPoqlM1MZb7ciBPpBK4v192d23p+vSeTVEXharYXy1bgdELVVp6TAWr65jXT2pqqibrgYQ4bVmnUqBxL0XTlA9m0TRF51KWAJIWUwfbJoRTPJlkkKAbWopweAXEogac4RlKfqsjvApeIcoDu4NuBt1miIXzNmzKDh0o+3gSZJs2Kz+KdOBCfxHvxRfw4kitnDq1nD3+3yJXx1uTT8yvf/5kV4J/Rw99ZIzVn1KYF1qqgPWZEr8Ir8ntPnh+s3GjM5pfFG/EV+l+LffERKwh737y3y7LxitnvIKfoYgLrfqngAfgkEO3pHlxFFZtjOlCp6/VZtfZtmhtg/8vo0O4QY4Ed3bMM821w6XOQjox0wVbMh/Af894GXCHEidI6GlzdHuyAiydnlHuJV6/FwapM1FSwfWSOYlw9xJjAyhn9xZhxN0IojpGtdY/iU2z1+Zy3/qL1HuZs/GuYt7lKle+LhF6berwnRed1RJHvwt9l2+XTvj3gNGmG42ZKpiruq3X4dg4ba9frlqhby4LG6SJdoiugmKdbtEhL1IQARc/oBb00Lhg3jbvFzR6rlFf8HP0xjMWfSXv0oQ==
AAAEjHicnVNNb9NAEJ2UACUU2nICcbFIixCgaN0e2nIBCYTKAdGmTVupjirb3YSl/pLthBYrf4ALR0CcQOKA+A9cuPAHOPQnII5F4sKBt2M3CEqCxFq2Z97OvHk7u+tEnkpSIfZLI8fKx0+cHD1VOT125uz4xOTYWhJ2Ylc23NAL4w3HTqSnAtlIVerJjSiWtu94ct3Zua3n17syTlQYrKZ7kWz6djtQLeXaKaCGZ1wzzK2JqqjNLMyL2QXjqGHWBI8qFWMpnCx9IIu2KSSXOuSTpIBS2B7ZlODZJJMERcCalNEjIDbF8BTHSOpRBfkd4BJRNtAdfNvwNgs0gK95E2ZwUcnDGyPToGnxWbwTB+KTeC++iB8DuTLm0Hr28HfyXBltjT89v/L9n1k+/ik9/JU1VHNKLZpnrQraI0b0Ktw8v/vk+cHKjfp0dlm8EV+h/7XYFx+xgqD7zX27LOuvmP0OcvIuxrDuFwoegE8C0Z7uwVVUsTimDZW6Xo9Va9+i633sfxlt2j3CmGODe5ZivgUufQ6SoZEO2PL5AP5j3lufKwQ4UVpHnatb/R1w8GSMci/x6rXYWJWBmgq2h8xhjKt/MMawMkYPGVPuRgDFEbK17mF8iq0en/PmX7Tew5yFfxXzFlep8H2R0GtRl/ck77yOyPMd+LtsO3zat/ucBk1x3FTBVMF9PbyUxmBjbaZmipq5LGiULtIlugKKObpFi7REDQhQ9Ixe0MvyhfLN8t38Zo+Uiit+jn4b5cWfxDX0/g==
ˆ
Gm
(
j
) =
8
>
>
>
>
>
<
>
>
>
>
>
:
j
1 
j

f
0:
l
o
f
0:
l
o
+
Gl
(
j
)
f
0:
l
o
<
j

f
0:
l
o
+ ˆ
fl
o
j
ˆ
fl
o
+
fl
o f
0:
l
o
+ ˆ
fl
o
<
j

f
0:
m
o
+ ˆ
fl
o fl
o
AAAF0HicnVNLTxRBEK5BV3F8AHox8dJxYYMgm148CKgJiSbowQjLM2FgMzP0LrPMi5nZFZxMjFf+gAdPkngw/gcvXrhq4oGfYDxi4sWD1TUDuDyWxJ7MdPVXVV99XdNt+LYVRpzvKh3nzucuXOy8pF6+cvVaV3fP9bnQawSmmDU92wsWDD0UtuWK2ciKbLHgB0J3DFvMG2uPpX++KYLQ8tyZaNMXS45ec62qZeoRQpUe5YFmiJrlxmK9QdBAomqrehRPJJXYSfrrd9gjNQsxsU6YqAxHnRVYiWm2WEeTpupyzMdsTPISpmkU9C80yCYqsU18hRbHw5MIBhlpqCbLcStlnQ0dcw3K1MwunEHTWs3Jwo5GDbFDSlUT7sr+1sk+7FSlO8+Lw6Mj/N4oO26UipxGHrIx6fUon0GDFfDAhAY4IMCFCG0bdAjxWYQScPARW4IY6ojoEODKohgBCaiY30BcYJSO6Bp+a7hazFAX15I3JAYTK9n4BpjJoI9/5x/5Ht/hn/gP/udUrpg4pJ5NnI00V/iVrq2b07/PzHJwjmD1MKut5giqMEJaLdTuEyJ3Yab5zVdv96bHyn1xgW/zn6j/Pd/lX3AHbvOX+WFKlN8R+xPMSbsYoPU8U/AC+QQiciV7MIBVNIqpoUpZLyHVcq3B3QPsfxl12DjGmGKn9yxCfxW55DkI20YayJb6XVy/pH/rUAUXT5TUUabq2sEfMPCJCaVe4iv3ouOuGNa00LYxsx3jzBHGAK2Y0H3GiLrhomIfs6XudnwWWQmd86UTtD5Dn4ZzHv0aVVHpvgjUq0GT/knaeRmR5hu43iDboNO+csDJoJfiejMmFe/r/qVkpxtzw8USL5amhvPjI9nN7YRbcBv6kes+jMNTmIRZMJVtZUf5qnzLlXMbude5N2loh5Ll3ICWkdv6C4TtZRM=
AAAFxXicnVNLTxRBEK5BV3FEAU8mXjoCG+SVXi8a1MREE/RghOWZ0LCZGXqXWebFzOwKTibGq3/AgydJPBj/gxcvXDXxwE8wHjHx4sHqmgFcFpbEnsx09VdVX31d020Gjh3FnO9pXefOFy5c7L6kX+65crW3r79nIfIboSXnLd/xwyXTiKRje3I+tmNHLgWhNFzTkYvmxiPlX2zKMLJ9by7eDuSKa9Q8u2pbRoxQpV+7J0xZs71EbjYIGkl1sW7EyVRaSdx0uH6LPdDzEAvrRKnOcNRZkZWYcOQmmjRVVxM+6WCSnzIhKOhfaJRNVRKH+IotjvsnEYwy0lBNV5NWyjobb3ONqtTcLp5B01rNzcOOR42zI0pdSG/tYOtkH3Wq0jfAJzgN1m6UcmMA8jHt92ufQcAa+GBBA1yQ4EGMtgMGRPgsQwk4BIitQAJ1RAwIcWVTjIQUdMxvIC4xykB0A781XC3nqIdrxRsRg4WVHHxDzGQwxL/zj3yf7/JP/Af/cypXQhxKzzbOZpYrg0rvm+uzv8/McnGOYf0oq6PmGKpwl7TaqD0gRO3CyvKbL9/uz06Wh5Ii3+E/Uf97vse/4A685i/rw4wsvyP2x5iTdTFE61mu4DnySUTUSvVgBKsIiqmhSlUvJdVqLWDsEPtfRgO22hgz7PSexeivIpc6B1HHSBPZMr+H6xf0b12q4OGJUjrKVF0c/gETn4RQ6iW+ai8G7ophTRttBzM7Mc4dYwzRSgg9YIypGx4qDjBb6e7EZ5OV0jlfOUHrU/QJnAfQL6iKTvdFol4BTfonWedVRJZv4nqLbJNO+9ohJ4NBihvMmXS8r6Xjt7PdWLg9UeITpRkO3XADbsIwUtyBh/AEpmEeLG1H29W+at8K5cJW4VV2s7u0/Ipfg5ZReP0Xn8Fjgw==
AAAFxXicnVPPT9RAFH5FV7GigCcTLxOBDYJsZvEgoCYmmqAHIyw/E7ps2jK7dOkv2u4KNo3x6j/gwZMkHoz/gxcvXDXxwJ9gPGLixYNvXgu4wC6J07Tz5nvvfe+b1xnDt60w4nxP6Tp3PnfhYvcl9XLPlau9ff09i6HXCEyxYHq2FywbeihsyxULkRXZYtkPhO4YtlgyNh5J/1JTBKHlufPRti/Kjl5zrapl6hFClX7lnmaImuXGYrNB0Eiiaut6FE8nldhJhuu32AM1CzGxTpioDEed5VmRabbYRJOm6mrMp2xM8hKmaRT0LzTKpiuxTXz5Fsf90whGGWmoJqtxK2WdjZ1wjcrUzM6fQdNazcnCjkeNsSNKVRPu2sHWyT7qVKVvgBfGJyf4nUl20igWOI0ByMaM1698Bg3WwAMTGuCAABcitG3QIcRnBYrAwUesDDHUEdEhwJVFMQISUDG/gbjAKB3RDfzWcLWSoS6uJW9IDCZWsvENMJPBEP/OP/J9vss/8R/8T1uumDiknm2cjTRX+JXeN9fnfp+Z5eAcwfpRVkfNEVRhgrRaqN0nRO7CTPObL9/uz02VhuI83+E/Uf97vse/4A7c5i/zw6wovSP2x5iTdjFA61mm4DnyCUTkSvZgBKtoFFNDlbJeQqrlWoPbh9j/MuqwdYIxxdr3LEJ/FbnkOQg7RhrIlvpdXL+gf+tQBRdPlNRRoura4R8w8IkJpV7iK/ei464Y1rTQtjGzE+P8McYArZjQA8aIuuGiYh+zpe5OfBZZCZ3z8ilan6JPw3kA/RpVUem+CNSrQZP+Sdp5GZHmG7jeItug0752yMlgkOIGMyYV7+vBpWTtjcXxQpEXirMcuuEG3IRhpLgLD+EJzMACmMqOsqt8Vb7lSrmt3Kv0Zncp2RW/Bi0j9/ovGopj4A==
AAAF0HicnVNLTxRBEK5BV3F8AHox8dJxYYMgm148CKgJiSbowQjLM2FgMzP0LrPMi5nZFZxMjFf+gAdPkngw/gcvXrhq4oGfYDxi4sWD1TUDuDyWxJ7MdPVXVV99XdNt+LYVRpzvKh3nzucuXOy8pF6+cvVaV3fP9bnQawSmmDU92wsWDD0UtuWK2ciKbLHgB0J3DFvMG2uPpX++KYLQ8tyZaNMXS45ec62qZeoRQpUe5YFmiJrlxmK9QdBAomqrehRPJJXYSfrrd9gjNQsxsU6YqAxHnRVYiWm2WEeTpupyzMdsTPISpmkU9C80yCYqsU18hRbHw5MIBhlpqCbLcStlnQ0dcw3K1MwunEHTWs3Jwo5GDbFDSlUT7sr+1sk+7FSlO8+Lw6Mj/N4oO26UipxGHrIx6fUon0GDFfDAhAY4IMCFCG0bdAjxWYQScPARW4IY6ojoEODKohgBCaiY30BcYJSO6Bp+a7hazFAX15I3JAYTK9n4BpjJoI9/5x/5Ht/hn/gP/udUrpg4pJ5NnI00V/iVrq2b07/PzHJwjmD1MKut5giqMEJaLdTuEyJ3Yab5zVdv96bHyn1xgW/zn6j/Pd/lX3AHbvOX+WFKlN8R+xPMSbsYoPU8U/AC+QQiciV7MIBVNIqpoUpZLyHVcq3B3QPsfxl12DjGmGKn9yxCfxW55DkI20YayJb6XVy/pH/rUAUXT5TUUabq2sEfMPCJCaVe4iv3ouOuGNa00LYxsx3jzBHGAK2Y0H3GiLrhomIfs6XudnwWWQmd86UTtD5Dn4ZzHv0aVVHpvgjUq0GT/knaeRmR5hu43iDboNO+csDJoJfiejMmFe/r/qVkpxtzw8USL5ameH58JLu5nXALbkM/ct2HcXgKkzALprKt7ChflW+5cm4j9zr3Jg3tULKcG9Ayclt/AYRNZRE=

View Slide

for DenseNet
18
• 層目にNet2DeeperNetが行われたときの
層目( )の入力は次となる：
• 追加された層の重みは次で初期化：
l
m
l < m
AAAEjHicnVPPa9RAFH5bo9a12taT4iW4rYjIMvFiKYKCIvUgtttuW2iWkqSz27H5RZJdW8P+A148qnhS8CD+D168+A946J8gHit48eA3L3HFrl3BCUne++a9733zZsaNfZVmQuxXxo4Zx0+cHD9VPT1x5uzk1PTEahp1E082vciPknXXSaWvQtnMVObL9TiRTuD6cs3duaPn13oySVUUrmR7sWwFTidUbeU5GaCmb940g82pmqgLHuawYZVGjcqxGE1XPpBNWxSRR10KSFJIGWyfHErxbJBFgmJgLcrpERCHEniKYyT1qYr8LnCJKAfoDr4deBslGsLXvCkzeKjk402QadKs+CzeiQPxSbwXX8SPI7ly5tB69vB3i1wZb04+Pb/8/Z9ZAf4Zbf/OGqk5ozbNsVYF7TEjehVekd978vxgeb4xm18Wb8RX6H8t9sVHrCDsffPeLsnGK2a/i5yiiwmsB6WCh+CTQLSne3AVVWyO6UClrtdn1dq36doA+19Gh3aHGAvs6J5lmG+DS5+DdGSkC7ZiPoT/mPc24AohTpTW0eDq9mAHXDw5o9xLvHotDlZloqaC7SNzFOPKIcYEVs7oL8aMuxFCcYxsrXsUn2Krz+e89Ret9zFn41/DvM1VqnxfJPTa1OM9KTqvI4p8F/4u2y6f9q0Bp0kzHDdTMlVxX63Dt3PYWL1et0TdWhI0ThfpEl0BxQ26TQu0SE0IUPSMXtBL44Jxy7hX3OyxSnnFz9Efw1j4CYiA9O4=
AAAEjHicnVNNb9NAEJ0UAyUU2nICcbFIixBC0aY90CIkkECoHBBt2rSV6qiy3U3Y1l+yndBi5Q9w4QiIE0gcEP+BCxf+AIf+BMSxSFw48HbsBkFJkFjL9szbmTdvZ3edyFNJKsR+aeSYcfzEydFT5dNjZ86OT0yOrSZhJ3Zlww29MF537ER6KpCNVKWeXI9iafuOJ9ecnTt6fq0r40SFwUq6F8mmb7cD1VKunQJqeOZN09+cqIjqzPycmJ03jxq1quBRoWIshpOlD2TRFoXkUod8khRQCtsjmxI8G1QjQRGwJmW0DcSmGJ7iGEk9KiO/A1wiyga6g28b3kaBBvA1b8IMLip5eGNkmjQtPot34kB8Eu/FF/FjIFfGHFrPHv5OniujzfGn55e//zPLxz+lR7+yhmpOqUVzrFVBe8SIXoWb53efPD9YvlGfzi6LN+Ir9L8W++IjVhB0v7lvl2T9FbPfRU7exRjWg0LBQ/BJINrTPbiKKhbHtKFS1+uxau1bdK2P/S+jTbtHGHNscM9SzLfApc9BMjTSAVs+H8B/zHvrc4UAJ0rrqHN1q78DDp6MUe4lXr0WG6syUVPB9pA5jHHlD8YYVsboIWPK3QigOEK21j2MT7HV43Pe/IvW+5iz8K9g3uIqZb4vEnot6vKe5J3XEXm+A3+XbYdP+1af06QpjpsqmMq4r4eX0hxsrM5Ua6JaWxI0ShfpEl0BxXW6TQu0SA0IUPSMXtBL44Jxy7iX3+yRUnHFz9Fvw1j4CQNJ9Us=
[O0, O1, . . . , Ol 1, Ol, Onew, Ol+1, . . . , Om 1
]
AAAE8nicnVPLThRBFL3gqDg+GHRj4qbjgDGKkxpcCC4MiS50YXgOkNDjpLspoJ1+pbtmADvzA/6Aia7QuDD6DW504Q+44BPUJSRuXHjqdjMGgTGxOzN176l7zj1VXWVHnpsoIXb6+k8UTp46PXCmePbc+QuDpaGLC0nYih1Zc0IvjJdsK5GeG8iacpUnl6JYWr7tyUW7eV/PL7ZlnLhhMK+2Iln3rbXAXXUdSwFqlJ6Ynh1bTtOYaqSiM6qHKgZzJVQJZ96tagZ72WD6llqP/TSQG5184uZBhg+GYcYs2yiVRWVsYlzcnjAOB9WK4KdM+TMdDvV9JJNWKCSHWuSTpIAUYo8sSvAuU5UERcDqlNJTIBbFyFyukdShIvgt4BJVFtAm/teQLedogFzrJqzgoJOHXwymQSPiq3gndsUX8V58E7+O1UpZQ/vZwmhnXBk1Bp9fnvv5T5aPUdH6H1ZPz4pWaZy9uvAeMaJX4WT89rMXu3N3Z0fSa+K1+AH/22JHfMIKgvae83ZGzr5i9QfgZLsYI3qcO5iCngSiM70HN9DF5Jo1uNT9Ouxa5yaNdrH/VbRo85Bihh2/Zwrzq9DS5yDpWWlDLZsPkG/wt/W5Q4ATpX3Mcnez+wVsvCmjvJf46bVYWJWBni5iD8xeivN/KcaIUkb3FRXvRgDHEdjady89l6MOn/P6EV4fYc7EWMa8yV2KfF8k/JrU5m+S7byuyPg28k2ObT7tK11Ng4a5bjhXKuK+7l9K4/hgYaxSFZXqjChP3stv7gBdoat0HVp3aJIe0jTV4OQzfadd2iuowsvCduFNVtrfl3Mu0YGn8OE3oZMY8A==
ˆ
F
AAAEmXicnVNNb9NAEJ0UAyV89IMLUi8RaRFCKNqUQ1sOqAiEChKiTZqmUh1VtrtJTR3bsjehxcof4MYJiZ5A4oD4D1y48Ac49CcgjkXiwoG3YzcISlKJtWzPvJ1583Z21w49N1ZCHORGThmnz5wdPZc/f+HipbHxicm1OOhEjqw5gRdE67YVS8/1ZU25ypPrYSSttu3Jur1zT8/XuzKK3cBfVXuhbLStlu82XcdSgOrmtqWSB73N8aIozS7Mi1sLheNGuSR4FCkby8FE7iOZtEUBOdShNknyScH2yKIYzwaVSVAIrEEJPQViUQTP5RhJPcojvwNcIsoCuoNvC95GhvrwNW/MDA4qeXgjZBZoRnwR78Wh+Cw+iK/i50CuhDm0nj387TRXhptjL65Uf5yY1cZf0fbvrKGaFTVpnrW60B4yolfhpPnd568Oq7crM8k18VZ8g/434kB8wgr87nfn3Yqs7DP7feSkXYxgPc4UPAGfBKI93YMbqGJyTAsqdb0eq9a+STf72P8yWrR7jDHFBvdMYb4JLn0O4qGRNtjSeR/+M97bNlfwcaK0jgpXN/s7YONJGOVe4tVrsbCqAmq6sD1kDmNc/YsxgpUwesSouBs+FIfI1rqH8bls9ficN/6h9SHmTPyLmDe5Sp7vi4Rek7q8J2nndUSab8PfZdvm077V5yzQNMdNZ0x53NejS1kYbKzNlsqiVF4RxcU72c0dpSm6StfBNUeLtETLVOO9eUmvad+YMu4aS8ajNHQkl+Vcpj+GUf0FVNv33g==
ˆ
F
[
x
,
y
,
n
] =
2
6
6
6
6
6
6
6
6
4
0 0 0
0 1 0
0 0 0
3
7
7
7
7
7
7
7
7
5
AAAFNXicnVPNbtNAEB6XQIv5aQsXJC4WaStUhWhTJNoigSqBEBwQ/W+lbBTZ7iY1dRyz3oQGyy/AC3DgVCQOiAfgxoULL8ChB24ICXHgUCQuHJgduy00JEistd6Zb2e++fbPCX0vUoztGgPHcsdPDA6dNE+dPnN2eGT03GrUbElXrLhNvynXHTsSvheIFeUpX6yHUtgNxxdrztYtPb/WFjLymsGy6oSi0rDrgVfzXFshVB3Z4Y6oe0EsHrUImUxMvmmr+E5S3i5YnYIVVKwblsl9UVNl08KWJdhS2p0kllImBFvMmsg654dIqQuhnjKJYCPjQZ9Lr76pKiahv8mpjuRZcWp2hl2dtbqNUpFRy0PW5pujxlvgsAFNcKEFDRAQgELbBxsi/MpQAgYhYhWI4SEiNkj0PIoRkICJ+S3EBUbZiG7hv45eOUMD9DVvRAwuVvKxS8y0YJx9YK/YHnvPXrMv7GdPrpg4tJ4Ojk6aK8Lq8NMLSz/+mdXAUcHmYVZfzQpqMENaPdQeEqJX4ab57SfP9pauL47HE+wF+4r6d9gue4crCNrf3ZcLYvE5sd/GnHQXJVr3MwUPkE8goj29B5NYhVNMHVXqegmp1j6HwgH2v4w2bHcxpljvPVM4X0MufQ+ivpEOsqXzAfqP6WwbVCHAG6V1LFJ1fnACDn4xobSX2PVabFyVhTU9tH3M7Me4fIRRohUTus+oaDcCVBxittbdj88jK6F7XvmL1ns4x3HM4zynKia9F4F6ObTpTNKd1xFpvoP+NtkO3faNA04LxihuLGPS73X/UVq9jdWpYokVSwtT+bmb2csdgotwCS4j1zTMwV2YhxVwjUHjinHNmM69yX3Mfcp9TkMHjCznPPzRct9+ARC2JUo=
フィルタサイズ
が3x3の場合
• nはチャネルのインデックス（ランダムに指定）
• 選ばれたチャネル以外は全てゼロ行列で初期化
Onew = ˆ
F([O0, . . . , Ol])
AAAE1nicnVPPaxNBFH6tUWv80VQvgpfFtFKlhEk92ApKURE9SH+3gU4Iu5tJsmZ/sTtNW5f1JoInbx48KQiK/4MXL568ifZPEI8VvHjwm9m0UmsiOMvuvvfNe9/75s2MFbpOLBnbHhg8lDt85OjQsfzxEydPDRdGTq/EwXpki2U7cIOoYpmxcB1fLEtHuqISRsL0LFesWu2ban61I6LYCfwluRWKqmc2fafh2KYEVCvcmK0l3DNlK/ISX2ykqXHN4C1TJrdTY9zgrhWZdttAEEsnDF4PZDyhPDc1eJTNXawViqw0OT3FLk8bB41yielRpO6YC0YG3hGnOgVk0zp5JMgnCdslk2I8a1QmRiGwKiV0H4hJETxHxwhKKY/8deACUSbQNr5NeGtd1IeveGPNYKOSizdCpkFj7BN7w3bYB/aWfWU/e3IlmkPp2cLfynJFWBt+cnbxxz+zPPwltX5n9dUsqUFTWqsD7aFG1CrsLL/z4NnO4tWFseQCe8m+Qf8Lts3eYwV+57v9al4sPNfst5CTdTGCda+rYBZ8AojyVA8uoQrXMU2oVPVSrVr5nCb2sP9lNGnzAGOG9e6ZxHwDXOocxH0jLbBl8z78Db23nq7g40QpHQu6Ot/bAQtPolHdS7xqLSZWZaCmA9tFZj/GpT8YI1iJRncZpe6GD8UhspXufnyOtlJ9zqt/0XoXcxz/Iua5rpLX90VAL6eO3pOs8yoiy7fgb2rb0qe9vsdp0KiOG+0y5XFfdy+l0dtYmSyVWak8z4oz17s3d4jO0XkaB9cVmqE7NEfLUPKaPtJn+pKr5B7mHuUeZ6GDA92cM7Rv5J7+AtQODtk=
ただし

View Slide

Experiments and Results
19
• 探索する構造は2種類
• Conv層，プーリング層，全結合層で構成される
スキップ結合なしのCNN
• DenseNetをベースとしたスキップ結合ありのCNN
• GeForce1080 GPUを5枚使用して実験
• NAS(800GPUs)と比べると，計算量に対して
強い制約を設けた状況

View Slide

(Image Datasets)
20
実験で使用したデータセット
名前訓練 / 検証 / テスト / 前処理などクラス数ドメイン
CIFAR-10
(C10)
45,000 / 5,000 / 10,000 /
標準化(平均0，分散1にスケール)
データ拡張(C10+と表記)，
10
一般画像
分類
SVHN
68,257 / 5,000 / 26,032 /
正規化(255で割る)
ベスト構造の学習：68,257 + 531,131
10
表札数字
分類
< Back to Alex Krizhevsky's home page
The CIFAR-10 and CIFAR-100 are labeled subsets of the 80 million tiny images dataset. They were collected by
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.
The CIFAR-10 dataset
The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are
50000 training images and 10000 test images.
The dataset is divided into five training batches and one test batch, each with 10000 images. The test batch contains
exactly 1000 randomly-selected images from each class. The training batches contain the remaining images in
random order, but some training batches may contain more images from one class than another. Between them, the
training batches contain exactly 5000 images from each class.
Here are the classes in the dataset, as well as 10 random images from each:
airplane
automobile
bird
cat
deer
dog
frog
horse
"3BOE$*'"3EBUBTFUT IUUQTXXXDTUPSPOUPFEVdLSJ[DJGBSIUNM
The CIFAR-10 and CIFAR-100 are labeled subsets of the 80 million t
The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 c
The dataset is divided into five training batches and one test batch, e
exactly 1000 randomly-selected images from each class. The training
random order, but some training batches may contain more images fr
Here are the classes in the dataset, as well as 10 random images fro
airplane
automobile
bird
cat
deer
dog
frog
horse
ship
truck
$*'"3BOE$*'"3EBUBTFUT
airplane
automobile
bird
cat
deer
dog
frog
horse
ship
truck
"3BOE$*'"3EBUBTFUT IUUQTXXXDTUPSPOUPFEVdLSJ[DJGBSIUNM
The CIFAR-10 and CIFAR-100 are labeled subsets of the 80 million ti
The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 c
The dataset is divided into five training batches and one test batch, e
exactly 1000 randomly-selected images from each class. The training
random order, but some training batches may contain more images fr
Here are the classes in the dataset, as well as 10 random images fro
airplane
automobile
bird
cat
deer
dog
frog
horse
ship
truck
The classes are completely mutually exclusive. There is no overlap b
includes sedans, SUVs, things of that sort. "Truck" includes only big t
These are the original, variable-resolution, color house-
bounding boxes, as shown in the examples images abov
illustration purposes. The bounding box information are
directly on the images in the dataset.) Each tar.gz file co
together with a digitStruct.mat file, which can be loaded
contains a struct called digitStruct with the same length
element in digitStruct has the following fields: name wh
corresponding image. bbox which is a struct array that c
digit bounding box in the image. Eg: digitStruct(300
digit bounding box in the 300th image.
Format 2: Cropped Digits: train_32x32.mat, test_32x
commercial use only)
5IF4USFFU7JFX)PVTF/VNCFST 47)/
%BUBTFU
airplane
automobile
bird
cat
deer
dog
frog
horse
ship
truck
The classes are completely mutually exclusive. There is no overlap between automobiles and trucks. "Automobile"
includes sedans, SUVs, things of that sort. "Truck" includes only big trucks. Neither includes pickup trucks.
CIFAR-10*1
SVHN*2
*1：https://www.cs.toronto.edu/~kriz/cifar.html *2：http://ufldl.stanford.edu/housenumbers/

View Slide

(Training Details)
21
• 層数・ユニット数：1層・50個
• 埋め込み層の出力：16次元
• 最適化法：Adam (学習率は小さめ)
• サンプル数：10
• 評価値：tan(検証データの精度 × π / 2)
• 60%→61%よりも90％→91%のほうが良い評価値
• ベースライン：指数移動平均（係数0.95）
Bi-directional LSTM (Encoder Net)

View Slide

(Training Details)
22
• 最適化法：Nesterov Momentum
• 初期学習率・慣性の強さ：0.02・0.9
• 学習率のスケジュール：
• Weight Decay：0.0001
• バッチサイズ：64
• エポック数：20
[Gastaldi 2017]の実装*3より
⌘init
AAAEqHicnVO9bhNBEB4HA8H8xAEKJBoLJwhFkbUOBQ5VJCigQDhOHAflLOvusnaO3J/u1ibJyS/AC1DQABJF4B1oaHgBijwCogwSDQXfzl2MIMRI7OnuZr6d+ebb2V0rdJ1YCXGQmziVP33m7OS5wvkLFy9NFacvr8VBP7Jl0w7cIFq3zFi6ji+bylGuXA8jaXqWK1vW9j093xrIKHYCf1XthrLtmT3f6Tq2qQB1ilcNqcxOYnim2oq8xPEdNRx2imVRWVisiduLpeNGtSJ4lCkb9WA694EM2qSAbOqTR5J8UrBdMinGs0FVEhQCa1NCT4GYFMFzOEbSkArI7wOXiDKBbuPbg7eRoT58zRszg41KLt4ImSWaFZ/FvjgUn8R78UX8OJErYQ6tZxd/K82VYWfq+bWV7//M8vBXtPUra6xmRV2qsVYH2kNG9CrsNH+w9+Jw5W5jNrkp3oiv0P9aHIiPWIE/+Ga/XZaNl8x+HzlpFyNYjzIFj8EngWhP92AOVQyO6UGlrjdk1do3aH6E/S+jSTvHGFPs5J4pzHfBpc9BPDbSAls678N/xnvrcQUfJ0rraHB1Y7QDFp6EUe4lXr0WE6sqoaYD20XmOMbVPxgjWAmjR4yKu+FDcYhsrXscn8PWkM95+y9aH2LOwL+MeYOrFPi+SOg1aMB7knZeR6T5Fvwdti0+7ZsjzhLNcNxMxlTAfT26lKWTjbWFSlVUqsvV8lItu7mTdJ1u0C1w3aElekB1akLJHr2ifXqXn8vX8638kzR0IpflXKHfRt76CQ7B/jM=
0
.
5
⌘
init (1
+
cos (
⇡T
current
/T
total))
AAAFBHicnVO7bhNBFL0OTgjmkQQaJJoRTlB4yIyDEAlVJCigQOSdSNnI2t2MnSH70u7YJKzcUvADFFQgUSDoEG0aGn4ApHwCogwSTQrOzG4cINhI7Gp37j1z77ln7sw4kScTxfluoe9YsX/g+OCJ0slTp88MDY+cXUrCZuyKRTf0wnjFsRPhyUAsKqk8sRLFwvYdTyw7m3f0/HJLxIkMgwW1HYk1324Esi5dWwGqDUe8cpNZQtm11PJttRH7qQykareZ5Ym6Gq+yq8xywyR3mRVJtnAY6zbjWAQ6/PqvsAqV7QFkzIplY0Nd7oysNlzmlYmpSX5jih01qhVunjLlz0w4Utghi9YpJJea5JOggBRsj2xK8K5SlThFwNYopUdAbIrhSRMjqE0l5DeBC0TZQDfxb8BbzdEAvuZNDIOLSh6+GJmMxvhn/obv8U/8Lf/K97typYZD69nG6GS5IqoNPTs//+OfWT5GRRuHWT01K6rTpNEqoT0yiF6Fm+W3njzfm789N5Ze4q/4N+h/yXf5R6wgaH13X8+KuReG/S5ysi7GsB7kCh6CTwDRnu7BFVSxTEwDKnW9tlGtfYuudbD/ZbRp6whjhnXvmcJ8HVz6HCQ9Ix2wZfMB/Mdmb31TIcCJ0jrmTHWrswMO3tSgppf49FpsrIqhpoTtIbMX48IfjDGs1KAHjMp0I4DiCNlady8+aay2Oedrf9F6H3MWxjLmLVOlZO6LgF6LWmZPss7riCzfgb9lbMec9vUOJ6NREzeaM5VwXw8uJetuLE1UqrxSna2WpyfzmztIF+gijYPrFk3TPZqhRSj5QvuF/sJA8WnxXfF98UMW2lfIc87Rb09x5yct/h+L
AAAE+XicnVM9bxQxEJ0LlxCOQBIqJBqLJCh86PAhIRAVEhRQIPKdSHF02t34Lib7pV3fkbC6loI/QEEFEgWCDtHS0PAHQMpPQJRBoknBs3e5QI5cJHa165nnmTfPY9uNfZVqzndKA8fKg0PHh09UTo6cOj06Nj6ylEatxJOLXuRHyYrrpNJXoVzUSvtyJU6kE7i+XHY375j55bZMUhWFC3o7lmuB0wxVQ3mOBlQfi3n1OhNSO/VMBI7eSIJMhUp3Okz4sqGna+wyE16UFi4TsWIL+7FeK0lkaMKv/gnrSDs+QMZEopob+mJ3ZPWxCV7l9mG9Rq0wJqh4ZqLx0kcStE4RedSigCSFpGH75FCKd5VqxCkGtkYZPQLiUAJP2RhJHaogvwVcIsoBuol/E95qgYbwDW9qGTxU8vElyGQ0xb/wN3yXf+Zv+Te+dyhXZjmMnm2Mbp4r4/ros7PzP4/MCjBq2tjP6qtZU4NuWq0K2mOLmFV4eX77yfPd+VtzU9kF/op/h/6XfId/wgrC9g/v9ayce2HZ7yIn72IC60Gh4CH4JBDjmR5cQhVhY5pQaep1rGrjC7rSxf6X0aGtHsYcO7xnGvMNcJlzkPaNdMGWz4fwH9u9DWyFECfK6Jiz1UV3B1y8mUVtL/GZtThYFUNNBdtHZj/GhQOMCazMor8Zte1GCMUxso3ufnzKWh17ztf+ofU+5gTGCcwLW6Vi74uEXkFtuyd5501Enu/C37K2a0/7epeT0aSNmyyYKrivtYO3s9dYulat8WptltMwnaPzNA2KG3Sb7tEMLULAV9orDZaGyk/L78rv85s9UCqu+Bn66yl/+AVDUx3J
AAAE+XicnVM9bxNBEB0HJwQTSEKFRLMiCQofMusgREKFBAUUiMT5lLKRdXdZO0vuS3drk3ByS8EfoKACiQJBh2hpaPgDIOUnIMog0aTg7Z7jACFG4k53O/N25s3b2V039lWqOd8p9B0r9g8cHzxROjl06vTwyOjQUho1E08uepEfJSuuk0pfhXJRK+3LlTiRTuD6ctndvG3ml1sySVUULujtWK4FTiNUdeU5GlBtJObl60xI7dQyETh6IwkyFSrdbjPhy7qerLDLTHhR2nGZiBVbOIj1mkkiQxN+9VdYR9rxATImEtXY0Be7I6uNjPHy1Mw0vzbDDhuVMrfPGHWe2Wi08IEErVNEHjUpIEkhadg+OZTiXaUKcYqBrVFGD4E4lMBTNkZSm0rIbwKXiHKAbuLfgLfaQUP4hje1DB4q+fgSZDKa4J/5a77LP/E3/CvfO5IrsxxGzzZGN8+VcW346dn5H//MCjBq2jjI6qlZU52mrVYF7bFFzCq8PL/1+Nnu/M3qRHaBv+TfoP8F3+EfsYKw9d17NSerzy37HeTkXUxg3e8oeAA+CcR4pgeXUEXYmAZUmnptq9r4gq50sf9ldGjrEGOOHd0zjfk6uMw5SHtGumDL50P4j+zeBrZCiBNldFRtddHdARdvZlHbS3xmLQ5WxVBTwfaR2Ytx4Q/GBFZm0X1GbbsRQnGMbKO7F5+yVtue87W/aL2HOYFxDPPCVinZ+yKhV1DL7kneeROR57vwt6zt2tO+3uVkNG7jxjtMJdzX/UvJjjaWpsoVXq7McRqkc3SeJkFxg27RXZqlRQj4QnuF/sJA8UnxbfFdfrP7Cp0rfoZ+e4rvfwK+DR4m
AAAFBHicnVO7bhNBFL0OTgjmkQQaJJoRTlB4yIyDEAlVJCigQOSdSNnI2t2MnSH70u7YJKzcUvADFFQgUSDoEG0aGn4ApHwCogwSTQrOzG4cINhI7Gp37j1z77ln7sw4kScTxfluoe9YsX/g+OCJ0slTp88MDY+cXUrCZuyKRTf0wnjFsRPhyUAsKqk8sRLFwvYdTyw7m3f0/HJLxIkMgwW1HYk1324Esi5dWwGqDUe8cpNZQtm11PJttRH7qQykareZ5Ym6Gq+yq8xywyR3mRVJtnAY6zbjWAQ6/PqvsAqV7QFkzIplY0Nd7oysNlzmlYmpSX5jih01qhVunjLlz0w4Utghi9YpJJea5JOggBRsj2xK8K5SlThFwNYopUdAbIrhSRMjqE0l5DeBC0TZQDfxb8BbzdEAvuZNDIOLSh6+GJmMxvhn/obv8U/8Lf/K97typYZD69nG6GS5IqoNPTs//+OfWT5GRRuHWT01K6rTpNEqoT0yiF6Fm+W3njzfm789N5Ze4q/4N+h/yXf5R6wgaH13X8+KuReG/S5ysi7GsB7kCh6CTwDRnu7BFVSxTEwDKnW9tlGtfYuudbD/ZbRp6whjhnXvmcJ8HVz6HCQ9Ix2wZfMB/Mdmb31TIcCJ0jrmTHWrswMO3tSgppf49FpsrIqhpoTtIbMX48IfjDGs1KAHjMp0I4DiCNlady8+aay2Oedrf9F6H3MWxjLmLVOlZO6LgF6LWmZPss7riCzfgb9lbMec9vUOJ6NREzeaM5VwXw8uJetuLE1UqrxSneXl6cn85g7SBbpI4+C6RdN0j2ZoEUq+0H6hvzBQfFp8V3xf/JCF9hXynHP021Pc+Qktrh+K
AAAE+XicnVM9bxQxEJ0LlxCOQBIqJBqLJCh86PAhIRAVEhRQIPKdSHF02t34Lib7pV3fkbC6loI/QEEFEgWCDtHS0PAHQMpPQJRBoknBs3e5QI5cJHa165nnmTfPY9uNfZVqzndKA8fKg0PHh09UTo6cOj06Nj6ylEatxJOLXuRHyYrrpNJXoVzUSvtyJU6kE7i+XHY375j55bZMUhWFC3o7lmuB0wxVQ3mOBlQfi3n1OhNSO/VMBI7eSIJMhUp3Okz4sqGna+wyE16UFi4TsWIL+7FeK0lkaMKv/gnrSDs+QMZEopob+mJ3ZPWxCV7l9mG9Rq0wJqh4ZqLx0kcStE4RedSigCSFpGH75FCKd5VqxCkGtkYZPQLiUAJP2RhJHaogvwVcIsoBuol/E95qgYbwDW9qGTxU8vElyGQ0xb/wN3yXf+Zv+Te+dyhXZjmMnm2Mbp4r4/ros7PzP4/MCjBq2tjP6qtZU4NuWq0K2mOLmFV4eX77yfPd+VtzU9kF/op/h/6XfId/wgrC9g/v9ayce2HZ7yIn72IC60Gh4CH4JBDjmR5cQhVhY5pQaep1rGrjC7rSxf6X0aGtHsYcO7xnGvMNcJlzkPaNdMGWz4fwH9u9DWyFECfK6Jiz1UV3B1y8mUVtL/GZtThYFUNNBdtHZj/GhQOMCazMor8Zte1GCMUxso3ufnzKWh17ztf+ofU+5gTGCcwLW6Vi74uEXkFtuyd5501Enu/C37K2a0/7epeT0aSNmyyYKrivtYO3s9dYulat8WptltMwnaPzNA2KG3Sb7tEMLULAV9orDZaGyk/L78rv85s9UCqu+Bn66yl/+AVDUx3J
AAAE+XicnVM9bxNBEB0HJwQTSEKFRLMiCQofMusgREKFBAUUiMT5lLKRdXdZO0vuS3drk3ByS8EfoKACiQJBh2hpaPgDIOUnIMog0aTg7Z7jACFG4k53O/N25s3b2V039lWqOd8p9B0r9g8cHzxROjl06vTwyOjQUho1E08uepEfJSuuk0pfhXJRK+3LlTiRTuD6ctndvG3ml1sySVUULujtWK4FTiNUdeU5GlBtJObl60xI7dQyETh6IwkyFSrdbjPhy7qerLDLTHhR2nGZiBVbOIj1mkkiQxN+9VdYR9rxATImEtXY0Be7I6uNjPHy1Mw0vzbDDhuVMrfPGHWe2Wi08IEErVNEHjUpIEkhadg+OZTiXaUKcYqBrVFGD4E4lMBTNkZSm0rIbwKXiHKAbuLfgLfaQUP4hje1DB4q+fgSZDKa4J/5a77LP/E3/CvfO5IrsxxGzzZGN8+VcW346dn5H//MCjBq2jjI6qlZU52mrVYF7bFFzCq8PL/1+Nnu/M3qRHaBv+TfoP8F3+EfsYKw9d17NSerzy37HeTkXUxg3e8oeAA+CcR4pgeXUEXYmAZUmnptq9r4gq50sf9ldGjrEGOOHd0zjfk6uMw5SHtGumDL50P4j+zeBrZCiBNldFRtddHdARdvZlHbS3xmLQ5WxVBTwfaR2Ytx4Q/GBFZm0X1GbbsRQnGMbKO7F5+yVtue87W/aL2HOYFxDPPCVinZ+yKhV1DL7kneeROR57vwt6zt2tO+3uVkNG7jxjtMJdzX/UvJjjaWpsoVXq7McRqkc3SeJkFxg27RXZqlRQj4QnuF/sJA8UnxbfFdfrP7Cp0rfoZ+e4rvfwK+DR4m
AAAFBHicnVO7bhNBFL0OTgjmkQQaJJoRTlB4yIyDEAlVJCigQOSdSNnI2t2MnSH70u7YJKzcUvADFFQgUSDoEG0aGn4ApHwCogwSTQrOzG4cINhI7Gp37j1z77ln7sw4kScTxfluoe9YsX/g+OCJ0slTp88MDY+cXUrCZuyKRTf0wnjFsRPhyUAsKqk8sRLFwvYdTyw7m3f0/HJLxIkMgwW1HYk1324Esi5dWwGqDUe8cpNZQtm11PJttRH7qQykareZ5Ym6Gq+yq8xywyR3mRVJtnAY6zbjWAQ6/PqvsAqV7QFkzIplY0Nd7oysNlzmlYmpSX5jih01qhVunjLlz0w4Utghi9YpJJea5JOggBRsj2xK8K5SlThFwNYopUdAbIrhSRMjqE0l5DeBC0TZQDfxb8BbzdEAvuZNDIOLSh6+GJmMxvhn/obv8U/8Lf/K97typYZD69nG6GS5IqoNPTs//+OfWT5GRRuHWT01K6rTpNEqoT0yiF6Fm+W3njzfm789N5Ze4q/4N+h/yXf5R6wgaH13X8+KuReG/S5ysi7GsB7kCh6CTwDRnu7BFVSxTEwDKnW9tlGtfYuudbD/ZbRp6whjhnXvmcJ8HVz6HCQ9Ix2wZfMB/Mdmb31TIcCJ0jrmTHWrswMO3tSgppf49FpsrIqhpoTtIbMX48IfjDGs1KAHjMp0I4DiCNlady8+aay2Oedrf9F6H3MWxjLmLVOlZO6LgF6LWmZPss7riCzfgb9lbMec9vUOJ6NREzeaM5VwXw8uJetuLE1UqrxSneXl6cn85g7SBbpI4+C6RdN0j2ZoEUq+0H6hvzBQfFp8V3xf/JCF9hXynHP021Pc+Qktrh+K
サンプルされたCNN
*3：https://github.com/xgastaldi/shake-shake/blob/master/train.lua#L206-L213

View Slide

(Explore Plain CNN Architecture Space)
23
• フィルタサイズ：1×1，3×3，5×5
• チャネル数：最小16〜最大512
16×(1, 2, 4, 6, 8, 16, 20, 24, 28, 32)
• ストライド：1
• 全結合層のユニット数：最小64〜最大1024
64×(1, 2, 4, 6, 8, 10, 12, 14, 16)
• 活性化関数：ReLU
• バッチ正規化，Dropout(SVHNのみ，rate 0.2)
サンプルされたCNNの具体的な設定

View Slide

24
• 入力する構造 (start point)：
• 学習を2ステップに分割
1. Net2Deeper 5回，Net2Wider 4回を実行
2. 300個サンプリング，ベストな構造を100エポック学習
1. Net2Deeper 5回，Net2Wider 4回を実行
2. 150個サンプリング，ベストな構造を300エポック学習
Step1
Step2
Table 1: Simple start point network. C(n, f, l) denotes a convolutional layer with n filters, filter size f and stride l; P(f, l, MAX)
and P(f, l, AVG) denote a max and an average pooling layer with filter size f and stride l respectively; FC(n) denotes a fully-
connected layer with n units; SM(n) denotes a softmax layer with n output units.
Model Architecture Validation Accuracy (%)
C(16, 3, 1), P(2, 2, MAX), C(32, 3, 1), P(2, 2, MAX), C(64, 3, 1),
P(2, 2, MAX), C(128, 3, 1), P(4, 4, AVG), FC(256), SM(10)
87.07
we use ReLU and batch normalization for each convolu-
tional or fully-connected layer. For SVHN, we add a dropout
layer after each convolutional layer (except the first layer)
and use a dropout rate of 0.2 (Huang et al. 2017).
Start with Small Network We begin the exploration on
C10+, using a small network (see Table 1), which achieves
87.07% accuracy in the held-out validation set, as the start
point. Different from (Zoph and Le 2017; Baker et al. 2017),
EAS is not restricted to start from empty and can flexibly
use any discovered architecture as the new start point. As
such, to take the advantage of such flexibility and also re-
duce the search space for saving the computational resources
and time, we divide the whole architecture search process
into two stages where we allow the meta-controller to take 5
steps of Net2Deeper action and 4 steps of Net2Wider action
in the first stage. After 300 networks are sampled, we take
the network which performs best currently and train it with
Table 2: Test error rate (%) comparison with CNNs that use
convolutional, fully-connected and pooling layers alone.
Model C10+ SVHN
human
designed
Maxout (Goodfellow et al. 2013)
NIN (Lin, Chen, and Yan 2013)
All-CNN (Springenberg et al. 2014)
VGGnet (Simonyan and Zisserman 2015)
9.38
8.81
7.25
7.25
2.47
2.35
-
-
auto
designed
MetaQNN (Baker et al. 2017) (depth=7)
MetaQNN (Baker et al. 2017) (ensemble)
EAS (plain CNN, depth=16)
6.92
-
4.89
4.23
-
2.06
1.83
1.73
space, in the second experiment, we use the top architec-
tures discovered in the first experiment, as the start points
to explore a larger architecture space on C10+ and SVHN.
This experiment on each dataset takes around 2 days on 5
GPUs.

View Slide

25
• ハンドメイドの構造よりも優れた構造
• C10+のベスト構造の重みを初期化＆
SVHNを40エポック学習させても良い結果に
• 5 GPUs × 2日
of the nth entry in [O0, O1, · · · , O
l
], we can set ˆ
F (using
the special case that knew
w
= knew
h
= 3 for illustration) as the
following
ˆ
F [
x, y, n
] =
2
4
0 0 0
0 1 0
0 0 0
3
5
,
(5)
while all other values in ˆ
F are set to be 0. Note that n can be
chosen randomly from {1, · · · , f0:
l
+1
o
} for each filter. After
all filters in the new layer are set, we can form an equivalent
random remapping function for all subsequent layers as is
done in Eq. (4) and modify their kernels accordingly.
In line with the previous work (Baker et al. 2017; Zoph and
Le 2017; Real et al. 2017), we apply the proposed EAS on
image benchmark datasets (CIFAR-10 and SVHN) to ex-
plore high performance CNN architectures for the image
classification task1. Notice that the performances of the final
(87.07)
(90.18)
(91.78)
(92.47)
(95.11)
100 epochs
training
300 epochs
training
Figure 4: Progress of two stages architecture search on C10+
in the plain CNN architecture space.
ional layer with n filters, filter size f and stride l; P(f, l, MAX)
h filter size f and stride l respectively; FC(n) denotes a fully-
th n output units.
Validation Accuracy (%)
MAX), C(64, 3, 1),
56), SM(10)
87.07
Model C10+ SVHN
human
designed
9.38
8.81
7.25
7.25
2.47
2.35
-
-
auto
designed
6.92
-
4.89
4.23
-
2.06
1.83
1.73

View Slide

Table 3: Test error rate (%) comparison with state-of-the-art architectures.
Model Depth Params C10+
human
designed
110
1202
16
28
1001
40
100
190
1.7M
10.2M
11.0M
36.5M
10.2M
1.0M
0.8M
25.6M
6.61
4.91
4.81
4.17
4.62
5.24
4.51
3.46
auto
designed
-
20
39
39
20
5.4M
2.5M
7.1M
37.4M
23.4M
5.40
6.01
4.47
3.65
4.23
(Further Explore Larger Architecture Space)
26
• 先程の実験で得られたベスト構造を入力と
して再度学習（5GPUs × 2日）
Q学習ベースの手法
MetaQNNよりも深い構造を
探索することが可能
スキップ結合付きの
構造に匹敵する性能
ional layer with n filters, filter size f and stride l; P(f, l, MAX)
h filter size f and stride l respectively; FC(n) denotes a fully-
th n output units.
Validation Accuracy (%)
MAX), C(64, 3, 1),
56), SM(10)
87.07
Model C10+ SVHN
human
designed
9.38
8.81
7.25
7.25
2.47
2.35
-
-
auto
designed
6.92
-
4.89
4.23
-
2.06
1.83
1.73
space, in the second experiment, we use the top architec-
tures discovered in the first experiment, as the start points
to explore a larger architecture space on C10+ and SVHN.
This experiment on each dataset takes around 2 days on 5
GPUs.
The summarized results of comparing with human-
designed and automatically designed architectures that use
a similar design scheme (plain CNN), are reported in Table
2, where we can find that the top model designed by EAS
on the plain CNN architecture space outperforms all similar
models by a large margin. Specifically, comparing to human-
designed models, the test error rate drops from 7.25% to

View Slide

(Comparison Between RL and Random Search)
27
• ランダムにネットワーク変換を施す探索方法
も考えられる（Random Search）
• Table 1の構造を入力としてNet2Deeper 5回，
Net2Wider 4回を実行した結果と比較
• EASのほうが
効率的に探索可能

View Slide

(Explore DenseNet Architecture Space)
28
• DenseNet-BCを入力にして構造探索
• DenseBlock + 特徴マップを縮小 (Bottleneck) +
チャネル数を削減 (Compression)
• 合計40(=L)層，DenseBlock内のConv層における
出力チャネル数40(=k; Growth rate)
• Bottleneck以外の層の出力チャネル数：
40, 44, 48, 52, 56, 60, 64
human
designed
28
1001
40
100
190
36.5M
10.2M
1.0M
0.8M
25.6M
4.17
4.62
5.24
4.51
3.46
auto
designed
-
20
39
39
20
5.4M
2.5M
7.1M
37.4M
23.4M
5.40
6.01
4.47
3.65
4.23
Figure 5: Comparison between RL based meta-controller
and random search on C10+.
Table 4: Test error rate (%) results of exploring DenseNet
architecture space with EAS.
Model Depth Params C10 C10+
DenseNet (L = 100, k = 24) 100 27.2M 5.83 3.74
DenseNet-BC (L = 250, k = 24) 250 15.3M 5.19 3.62
DenseNet-BC (L = 190, k = 40) 190 25.6M - 3.46
NAS (post-processing) 39 37.4M - 3.65
EAS (DenseNet on C10) 70 8.6M 4.66 -
EAS (DenseNet on C10+) 76 10.7M - 3.44
action. The result is reported in Figure 5, which
the RL based meta-controller can effectively fo
right search direction, while the random search c
plot), and thus find high performance architectur
ficiently than random search.
Explore DenseNet Architecture Space
We also apply EAS to explore the DenseNet a
space. We use the DenseNet-BC (L = 40, k
the start point. The growth rate, i.e. the width o
bottleneck layer is chosen from {40, 44, 48, 52, 5
and the result is reported in Table 4. We find
plying EAS to explore the DenseNet architecture
achieve a test error rate of 4.66% on C10, bett
best result, i.e. 5.19% given by the original Dens
having 43.79% less parameters. On C10+, we ac
error rate of 3.44%, also outperforming the best
3.46% given by the original DenseNet while havi
less parameters.
Conclusion
In this paper, we presented EAS, a new fram
パラメータ数削減＆
性能向上

View Slide

Conclusion
29
• 適切なネットワーク変換を探索するEASを提案
• function-preservingによって重みを再利用
することが可能
• 他の手法に比べて効率的かつ優れた構造の
探索を実現
• ネットワークの大きさと精度のバランスを保つ
構造探索とより多くの変換操作の探求が課題

View Slide

参考文献
30
• [Chen et al. 2016] T. Chen, I. Goodfellow, and J. Shlens, “Net2Net: Accelerating Learning via
Knowledge Transfer,” in Proceedings of 4th International Conference on Learning
Representations (ICLR’16), 2016.
• [Zoph & Le, 2017] B. Zoph and Q. V Le, “Neural Architecture Search with Reinforcement
Learning,” in Proceedings of 5th International Conference on Learning Representations
(ICLR'17), 2017.
• [Real et al., 2017] E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q. Le, and A.
Kurakin, “Large-Scale Evolution of Image Classifiers,” in Proceedings of the 34th
International Conference on Machine Learning, vol. PMLR 70, pp. 2902–2911, Mar. 2017.
• [Huang et al., 2017] G. Huang, Z. Liu, L. v. d. Maaten, and K. Q. Weinberger, “Densely
Connected Convolutional Networks,” in 2017 IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), 2017, pp. 2261–2269.
• [Baker et al., 2017] B. Baker, O. Gupta, N. Naik, and R. Raskar, “Designing Neural Network
Architectures Using Reinforcement Learning,” in Proceedings of 5th International Conference
on Learning Representations (ICLR’17), 2017.
• [Gastaldi 2017] X. Gastaldi, “Shake-Shake regularization,” Preprint arXiv
1705.07485v2, May 2017.

View Slide

論文紹介：Efficient Architecture Search by Network Transformation [Cai et al., AAAI-2018]

論文紹介：Efficient Architecture Search by Network Transformation [Cai et al., AAAI-2018]

S.Shota

More Decks by S.Shota

Other Decks in Research

Featured

Transcript