Share
Embed
Generalized ODIN
Scatter Lab Inc.
April 24, 2020
Transcript
31 slides
-
Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
ݾର! 1. ਃড (Abstract) 2. ߓ҃ध (Background) :Out-of-Distribution Detection పझח ޖੋо? 3. ࢶ೯ োҳ (Previous Work) 4. ޙઁ (Problem Setting) 5. Ӕ ߑߨ (Approach) 6. प (Experiment) 7. Ѿҗ (Results) 8. ࠙ࢳ (Analysis) -
ਃড Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution
Data • ࠄ ֤ޙীࢲח ण ؘఠ৬ زੌೞ ঋ ࠙ನ ੑ۱ ٜযৡ ࢚ടਸ хೞח OoD(Out-of-Distribution) Detection ޙઁܳ ಽҊ פ. • ӝઓ Ӕ ߑधٜ “In-Distribution”җ “Out-of-Distribution" ف ؘఠ ݽف ण ژח ೞಌۄఠ ౚী ࢎਊೞणפ. • ࠄ ֤ޙীࢲח OoD ؘఠܳ ੌ ࢎਊೞ ঋח ࢚ടীࢲ ޙઁܳ ೧ѾೞҊ פ. • ೧Ѿ ߑߨਵ۽ࢲ, ഛܫ Ӕਸ ߄ఔਵ۽ ೞח “Decomposed Confidence” ۄח ߑߨਸ ઁউפ. • पਸ ా೧ OoD ؘఠܳ ࢎਊೞৈ णೠ baseline ݽ؛ٜ ࠁ ࣻೠ ࢿמਸ Ѣل ࣻ ਸ ࠁৈસפ. -
Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
Out-of-Distribution(OoD) Detection ۆ? - णػ ݽ؛ਸ प ࢲ࠺झী ߓನೞݶ, णदী ࠁ ޅ೮؍ ੑ۱ٜ(=Out-of-Distribution)ਸ ࠁѱ ؽ - ӘԈ ࠁ ޅೠ ੑ۱ী ೧ࢲ ML ݽ؛ٜ ҭ ੜޅػ ز ژח ஏਸ ೣ - ೞ݅ ݽ؛ ѱ ੜޅػ ੑ۱ੋ ઁ۽ ػ ੑ۱ੋ ౸ױೡ ࣻ ח מ۱ হ - ܳ ٜযࢲ, धਸ ҳ࠙ೞח image classification model Ҋ ೮ਸ ٸ, Airbnb ߑ ࢎ ٜযয়ݶ যڌѱ ؼө? ಃா? ࢎҗ? - ܽ ۠ Ѿҗٜ ஏغӡ ߄ۄ ঋҊ, “ஏೡ ࣻ হ”ਸ ߜӡ ਗೠ -
Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
Out-of-Distribution(OoD) Detection ۆ? - ਤীࢲ ࢸݺೠ दܳ ࠁ ҳੋ ۨ߰ীࢲ ޙઁܳ ೠݶ - In-Distribution(Nѐ class) ؘఠ۽ णػ Multi-Class Classification ݽ؛ ਸ ٸ - పझ ؘఠࣇ যڃ ੑ۱ xо In-Distribution ী ࣘ೧ݶ x ഛೠ classܳ ஏೞӝ - xо In-Distribution ী ࣘ೧ ח ח Ѫ, ID ղ যڃ classী ೧ ػח Ѫ - పझ ؘఠࣇ যڃ ੑ۱ x*о In-Distribution ী ࣘ೧ ঋݶ, OoD ੑ۱ਸ хೞח Ѫ - x*о In-Distribution ী ࣘ೧ ঋח Ѫ, ID ղ যڃ classীب ೧غ ঋח ח Ѫ -
Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
ޙणפ ޙ! - Q. ID, OOD ܳ ҳ࠙ೞח binary classification model ਸ ٜ݅ݶ غחѢ ইצоਃ? - A. উఋӰѱب Ӓ ߑߨীח ݻо ޙઁ ઓפ. 1. classification णী ࢎਊೡ OOD ؘఠܳ ID ؘఠ ѐࣻ ݅ఀ OOD ҕрীࢲ sampling ೞݶ OOD ҕр ܳ അೞ ޅೞח ޙઁо णפ. ౠ OOD ҕр ݶ ࣻ۾ ޙઁח ؊ ఁפ. 2. sampling ߑߨী ٮۄࢲ ࢠ݂ػ ؘఠী bias о Ѧܾࣻ णפ. 3. ࠗ࠙ పझীࢲח IDژח OOD ޙ ѐࣻо ೠੑפ. (ܻ ӝמച ؘఠࣇب Ӓۧભ ƕƕ) -
“A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural
Networks”, 2017 ICLR 1. Out-of-distribution Detectionਸ ܘ ୭ୡ ֤ޙ 2. Out-of-distribution Detectionী ೠ ޙઁ ৬ ಣо ߑߨ, рױೠ ߬झۄੋ ١ਸ ઁद 3. Maximum Softmax Probability ч thresholdࠁ ݶ in-distribution, ਵݶ out-of-distribution sampleۄҊ ࠙ܨ -> ੌઙ max(softmax(p_i)) == confidence https://hoya012.github.io/blog/anomaly-detection-overview-2/ -
“Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks”,
2018 ICLR 1. ण դ neural networkী যځೠ ୶о ण হ рױೞݶࢲ ബҗਵ۽ out-of-distribution sampleٜਸ ইյ ࣻ . 2. Softmax Temperature scaling : in-distribution sampleҗ out-of-distribution sample softmax scoreܳ ࢲ۽ ؊ ݣযѱ ೞח ೡਸ ೣ. Tчਸ ੜ ࢶೞח Ѫ ਃೣ (T=1000 ࢎਊ) 3. Input preprocessing: ੌઙ Augmentation (Image Perturbation) 4. Baselineਸ Ѻର۽ ٮجܻݴ Out-of-distribution োҳ ࢿ оמࢿਸ ࠁৈળ ֤ޙ 5. ࠄ ֤ޙীࢲח োҳীࢲ ઁউೠ ߑߨਸ OoD ؘఠ হ ࢎਊೡ ࣻ ب۾ ѐࢶೣ https://hoya012.github.io/blog/anomaly-detection-overview-2/ ODIN -
- ৈ۞ োҳ [36, 29, 13]ীࢲ যڃ ੑ۱ٚ softmax
Ѿҗо ਵ۽ ֫ confidence ܳ ࠁח Ѿҗо ҙغ. - ܳ ٜয MNIST ࠙ܨ ݽ؛ী Gaussian Noise ܳ ੑ۱ਵ۽ חؘ, 91% probਵ۽ যڃ ېझо ஏغ؊ۄ - ೧ োҳٜ ۞ೠ Ѿҗо smooth ೠ दೣࣻ৬ ਬࢎೠ softmax ౠࢿ ٸޙۄҊ ೠ - softmaxח যڃ ೠ classܳ ࠙ݺೞѱ दೞח ౠࢿਸ ыҊ . - ٮۄࢲ softmax Ѿҗ۽ uniform ೠ distribution ইצ spiky (ࡶೠ) distribution ٜ݅য Ҋ ݈. - ܳ ೧Ѿೞӝ ਤ೧ softmax temperature scaling э ߑߨ ࢎਊغ. -> ખ؊ uniform ೞѱ غب۾ Over Confidence problem p(y|x) ܻо ण दఅ classifier: ੑ۱ x ী ೠ class probability Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data -
The Decomposed Confidence Score - ೞ݅ ࠄ ֤ޙ ח softmax
ޙઁ ࠁח ݽ؛ ۱чী ؊ ޙઁо ח Ѫী ୡਸ ݏ୶Ҋ ҒҒ ࢤп೧ ࠁও. - Ӓ җীࢲ overconfidence о غח ਗੋ softmax о ইצ ݽ؛ ۱чਸ ࢸ҅ೠ ࠗ࠙ী ח Ѫਸ ঌѱ ؽ p(y|x) ܻо ण दఅ classifier: ੑ۱ x ী ೠ class probability بݫੋী ೠ condition ഃ غয ঋ ݽٚ ੑ۱ x ∈ d in t "In-Distribtuion ী ࣘೞח ؘఠ"ۄח о ನೣغয . Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data -
ೞ݅ ܻח ੑ۱ਵ۽ In-Distributionҗ Out-of-Distribution ੋ ࢚ടਸ ݽف ܖӝ ٸޙী,
domain ী ೠ conditionalೞ ѱ output distribution ׳ۄઉঠ ೣ. Ӓېࢲ ݺदਵ۽ domain ী ೠ ઑѤࠗ ഛܫਸ ҳೞب۾ ਤ धਸ ߸҃ೣ The Decomposed Confidence Score - ೞ݅ ࠄ ֤ޙ ח softmax ޙઁ ࠁח ݽ؛ ۱чী ؊ ޙઁо ח Ѫী ୡਸ ݏ୶Ҋ ҒҒ ࢤп೧ ࠁও. - Ӓ җীࢲ overconfidence о غח ਗੋ softmax о ইצ ݽ؛ ۱чਸ ࢸ҅ೠ оী ח Ѫਸ ঌѱ ؽ p(y|x) ܻо ण दఅ classifier: ੑ۱ x ী ೠ class probability بݫੋী ೠ condition ഃ غয ঋ ݽٚ ੑ۱ x ∈ d in t "In-Distribtuion ী ࣘೞח ؘఠ"ۄח о ನೣغয . p(y|d in , x) domainਸ Ҋ۰ೞח णػ classifier : ੑ۱ x ী ೠ class probability Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data -
The Decomposed Confidence Score p(y|d in , x) domainਸ Ҋ۰ೞח
णػ classifier : ੑ۱ x ী ೠ class probability p(y|d in , x) = p(y, d in |x) p(din |x) ਤ ध ઑѤࠗ ഛܫী ٮۄࢲ җ э ѐೡ ࣻ . ܻח ѐػ धਸ ా೧ࢲ ৵ overconfidence о ߊࢤ೮ח ঌ ࣻ . Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data -
ઁݾ ఫझ The Decomposed Confidence Score p(y|d in , x)
domainਸ Ҋ۰ೞח णػ classifier : ੑ۱ x ী ೠ class probability p(y|d in , x) = p(y, d in |x) p(din |x) ਤ ध ઑѤࠗ ഛܫী ٮۄࢲ җ э ѐೡ ࣻ . ܻח ѐػ धਸ ా೧ࢲ ৵ overconfidence о ߊࢤ೮ח ঌ ࣻ . ܳ ٜয xо Out-of-Distribution ীࢲ sampling غҊ ೡ ٸ, p(y, d in |x) ծਸ чਸ ыਸ Ѫ. (: 0.09) p(d in |x) द ծਸ чਸ ыਸ Ѫ. (: 0.1) xо in-distribtution ੌ ഛܫਸ աఋղח y class ഛܫҗ xо in-distribution ੌ ഛܫ Ѿ࠙ನ -
ઁݾ ఫझ The Decomposed Confidence Score p(y|d in , x)
domainਸ Ҋ۰ೞח णػ classifier : ੑ۱ x ী ೠ class probability p(y|d in , x) = p(y, d in |x) p(din |x) ਤ ध ઑѤࠗ ഛܫী ٮۄࢲ җ э ѐೡ ࣻ . ܻח ѐػ धਸ ా೧ࢲ ৵ overconfidence о ߊࢤ೮ח ঌ ࣻ . ܳ ٜয xо Out-of-Distribution ীࢲ sampling غҊ ೡ ٸ, p(y, d in |x) ծਸ чਸ ыਸ Ѫ. (: 0.09) p(d in |x) द ծਸ чਸ ыਸ Ѫ. (: 0.1) p(y|d in , x) = p(y, d in |x) p(din |x) = 0.09 0.1 = 0.9 xо in-distribtution ੌ ഛܫਸ աఋղח y class ഛܫҗ xо in-distribution ੌ ഛܫ Ѿ࠙ನ xо out-of-distribution ীࢲ ࡳഊ ীب Ѿҗਵ۽ যڃ class confidence ח ݒ ֫ѱ աৡח Ѫਸ ঌ ࣻ -
The Decomposed Confidence Score p(y|d in , x) = p(y,
d in |x) p(din |x) - ਤ৬ э overconfidence ޙઁܳ ೧Ѿೞח о ҙੋ ߑߨ y(ېझ), d(بݫੋ) ۨ࠶ਸ ਊ೧ࢲ ف ݽ؛ਸ supervised-learning ਵ۽ णೞח Ѫ. - Ӓ۞ݶ classب ஏೡ ࣻ Ҋ, threshold ܳ ਊ೧ࢲ OoD द ࠙߹ೡ ࣻ . - ೞ݅ ߑߨ OoD ؘఠࣇ ਃೠ ߑߨ. ٸޙী ޙઁ оী ݏ ঋח. - = Out-of-Distribution ীࢲ ࢠ݂ ೞݶ ੜ উغҊ ~ ١١ p(y, d in |x) p(d in |x) arg max i (p(y i , d in |x)) p(d in |x) Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data -
- ѾҴ supervised learning হ ف ഛܫ ݽ؛ਸ णೞח
Ѫ Ӕࠄਵ۽ ࠛоמೣ. - ۠ ࢚ടীࢲח “self-supervised learning” ژח “unsupervised-learning” Ӕ ߑߨਸ ࢤп೧ ࠅ ࣻ . - ਤ৬ э Ӕ ߑߨਸ ࢎਊೞ۰ݶ, ݽ؛ীѱ “ࢎ ध" ژח “о”ਸ ࠗৈ೧ ঠ ೣ - ࠄ ֤ޙীࢲח ࠙/࠙ݽ ҳઑ धਸ classifierܳ ٜ݅ٸ ܻо णೞҊ ೞח పझ ࢎ ࠁ۽ ח ߑधਸ ࢎਊೞ. - ٍীࢲ ҅ࣘ ܖѷ݅ ਤ धҗ э ҳઑח, class probability confidence ܳ decompose ೡ ࣻ ח מ۱ਸ ࠗৈೠ The Decomposed Confidence Score p(y|d in , x) = p(y, d in |x) p(din |x) p(y, d in |x) p(d in |x) class_prob Decomposed confidence #1 Decomposed confidence #2 Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data -
The Decomposed Confidence Score p(y|d in , x) = p(y,
d in |x) p(din |x) Decomposed confidence #1 Decomposed confidence #2 Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data -
The Decomposed Confidence p(y|d in , x) = p(y, d
in |x) p(din |x) f i (x) = h i (x) g (x) Cro ssEn tro py(so ftmax(f i (x))) - ؘఠо In-distribution ࠙ನܳ ରೞח ҃ Multi-class classification ण p(y, d in |x) = h i (x) − > min imize g (x) − > min imize ৵ջೞݶ p(y, d in |x) ܳ ѾҴ ఃਕঠ ೞӝ ٸޙ p(y, d in |x) = h i (x) − > g reato r g (x) − > n o t − min imize ৵ջೞݶ p(y, d in |x) ఃח ߑೱਵ۽ h(x)о ೞҊ - ؘఠо In-distribution ݆ ࠙ನܳ ରೞח ҃ p(y, d in |x) ఃب۾ ण೧ঠ ೣ - ण: dataо in-distribution ੋ ҃ীח ೦࢚ Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data -
The Decomposed Confidence p(y|d in , x) = p(y, d
in |x) p(din |x) f i (x) = h i (x) g (x) g_x = sigmoid(BatchNorm(linear(backbone))) h_x_i = linear(backbone) h_x_E = -1 * L2(backbone, w_i) h_x_C = CosineSim(backbone, w_i) h_i(x)ח class ࠙ܨӝ g(x)ח in-distribution ੌ ഛܫ regressor पਸ ా೧ࢲ 3о ݽ؛ যڃ Ѫ ઁੌ જ ࠺Ү Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data -
The Decomposed Confidence p(y|d in , x) = p(y, d
in |x) p(din |x) f i (x) = h i (x) g (x) S DeCo n f (x) = max i h i (x) o r g (x) pred label = arg max i f i (x) o r arg max i h i (x) Out-of-Distribution Score Classification DeConf-I, DeConf-E, and DeConf-C ח h_i(x)ܳ ࢎਊೠ Ѫ যڃ Ѫਸ ࢶఖೡ ח पਸ ా೧ࢲ ঌই ࠁب۾ ೞ Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data -
Dataset ID Dataset - CVHN - CIFAR - 10 -
CIFAR - 100 OOD Dataset - TinyImageNet - LSUN - iSUN - Uniform Random Image - Gaussian Random Image Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data -
Metric ࢸݺ Area Under - Receiver operating characteristic curve AUROC
FPR: Negative ੋؘ Positive ۽ ஏೠ ࠺ਯ / 1- OOD recall TPR: Positive ੋؘ Positive ۽ ஏೠ ࠺ਯ / ID-recall TNR@TPR95 True Negative Rate @ True Positive Rate 95 Positive ੋؘ Positive ۽ ஏೠ ࠺ਯ 95%ੌٸ threshold ীࢲ Negative ੋؘ Negative ۽ ஏೠ ࠺ਯ , ID-recall == 0.95 о غח threshold ীࢲ OOD-recall ? Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data -
4ѐ OoD method ࢿמ ࠺Ү 4о method ݽف OoD ؘఠܳ
ࠁ ঋח оೞী evaluation ೠ Ѿҗ ण ؘఠ(ID) णؘఠ x OoD ࢿמ ഛੋਊ Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data -
Decomposed Confidence ߑध ࠺Ү Generalized ODIN: Detecting Out-of-distribution Image without
Learning from Out-of-distribution Data -
Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
ৈӝࢲ ഛੋ೧ ࠅ ࣻ ח അ࢚ - (c)৬ (d)ח ׀ীڸѱ ੜ clustering غয ח Ѫ. - (e)ח ਵ۽ score о ؘ֫, ח equeation4 ীࢲ ܟ؍ overconfidence അ࢚ ٸޙ -
хࢎפ✌ ୶о ޙ ژח ҾӘೠ ݶ ઁٚ ইې োۅ۽
োۅ ࣁਃ! ӣળࢿ (ML Research Scientist) [email protected] Facebook. @codertimo