Generalized ODIN

Transcript

1. (FOFSBMJ[FE
   0%*/
   
   %FUFDUJOH
   0VUPGEJTUSJCVUJPO
   *NBHF
   XJUIPVU
   -FBSOJOH
   GSPN
   0VUPGEJTUSJCVUJPO
   %BUB ӣળࢿ
   .BDIJOF
   -FBSOJOH
   3FTFBSDI
   4DJFOUJTU
2. Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data ݾର! 1. ਃড (Abstract) 2. ߓ҃૑ध (Background) :Out-of-Distribution Detection పझ௼ח ޖ঺ੋо? 3. ࢶ೯ োҳ (Previous Work) 4. ޙઁ ੿੄ (Problem Setting) 5. ੽Ӕ ߑߨ (Approach) 6. प೷ (Experiment) 7. Ѿҗ (Results) 8. ࠙ࢳ (Analysis)
3. ਃড Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data • ࠄ ֤ޙীࢲח ೟ण ؘ੉ఠ৬ زੌೞ૑ ঋ਷ ࠙ನ੄ ੑ۱੉ ٜযৡ ࢚ടਸ х૑ೞח 
 OoD(Out-of-Distribution) Detection ޙઁܳ ಽҊ੗ ೤פ׮. • ӝઓ੄ ੽Ӕ ߑधٜ਷ “In-Distribution”җ “Out-of-Distribution" ف ؘ੉ఠ ݽف ೟ण ژח ೞ੉ಌ౵ۄ޷ఠ ౚ׬ী ࢎਊೞ৓णפ׮. • ࠄ ֤ޙীࢲח OoD ؘ੉ఠܳ ੌ੺ ࢎਊೞ૑ ঋח ࢚ടীࢲ ੉ ޙઁܳ ೧ѾೞҊ੗ ೤פ׮. • ೧Ѿ ߑߨਵ۽ࢲ, ഛܫ੸ ੽Ӕਸ ߄ఔਵ۽ ೞח “Decomposed Confidence” ۄח ߑߨਸ ઁউ೤פ׮. • प೷ਸ ా೧ OoD ؘ੉ఠܳ ࢎਊೞৈ ೟णೠ baseline ݽ؛ٜ ࠁ׮ ਋ࣻೠ ࢿמਸ Ѣل ࣻ ੓঻਺ਸ ࠁৈસפ׮.
4. ߓ҃ ૑ध / Background ઁݾ ఫझ౟
5. Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data Out-of-Distribution(OoD) Detection ੉ۆ? - ೟णػ ݽ؛ਸ प ࢲ࠺झী ߓನೞݶ, ೟णदী ࠁ૑ ޅ೮؍ ੑ۱ٜ(=Out-of-Distribution)ਸ ࠁѱ ؽ - ૑ӘԈ ࠁ૑ ޅೠ ੑ۱ী ؀೧ࢲ ML ݽ؛ٜ਷ ҭ੢൤ ੜޅػ ز੘ ژח ৘ஏਸ ೣ - ೞ૑݅ ݽ؛਷ ੉ѱ ੜޅػ ੑ۱ੋ૑ ઁ؀۽ ػ ੑ۱ੋ૑ ౸ױೡ ࣻ ੓ח מ۱੉ হ਺ - ৘ܳ ٜযࢲ, ਺धਸ ҳ࠙ೞח image classification model ੉ ੓׮Ҋ ೮ਸ ٸ, 
 Airbnb ߑ ࢎ૓੉ ٜযয়ݶ যڌѱ ؼө? ಃா੉௼? ࢎҗ? - ਋ܽ ੉۠ Ѿҗٜ੉ ৘ஏغӡ ߄ۄ૑ ঋҊ, “৘ஏೡ ࣻ হ਺”ਸ ߜӡ ਗೠ׮
6. Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
7. Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data Out-of-Distribution(OoD) Detection ੉ۆ? - ਤীࢲ ࢸݺೠ ৘दܳ ࠁ׮ ҳ୓੸ੋ ۨ߰ীࢲ ޙઁܳ ੿੄ೠ׮ݶ - In-Distribution(Nѐ੄ class) ؘ੉ఠ۽ ೟णػ Multi-Class Classification ݽ؛੉ ੓ਸ ٸ - పझ౟ ؘ੉ఠࣇ੄ যڃ ੑ۱ xо In-Distribution ী ࣘ೧੓׮ݶ x੄ ੿ഛೠ classܳ ৘ஏೞӝ - xо In-Distribution ী ࣘ೧ ੓ח ׮ח Ѫ਷, ID ղ੄ যڃ classী ೧׼ ػ׮ח Ѫ - పझ౟ ؘ੉ఠࣇ੄ যڃ ੑ۱ x*о In-Distribution ী ࣘ೧੓૑ ঋ׮ݶ, OoD ੑ۱੐ਸ х૑ೞח Ѫ - x*о In-Distribution ী ࣘ೧੓૑ ঋ׮ח Ѫ਷, ID ղ੄ যڃ classীب ೧׼غ૑ ঋח ׮ח Ѫ
8. Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data ૕ޙ੓णפ׮ ૕ޙ! - Q. ID, OOD ܳ ҳ࠙ೞח binary classification model ਸ ٜ݅ݶ غחѢ ইצоਃ? - A. উఋӰѱب Ӓ ߑߨীח ݻо૑ ޙઁ੼੉ ઓ੤೤פ׮. 1. classification ೟णী ࢎਊೡ OOD ؘ੉ఠܳ ID ؘ੉ఠ ѐࣻ ݅ఀ OOD ҕрীࢲ sampling ೞݶ OOD ҕр ੹୓ܳ ׮ ಴അೞ૑ ޅೞח ޙઁо ੓णפ׮. ౠ൤ OOD ҕр੉ ௼ݶ ௿ ࣻ۾ ੉ ޙઁח ؊ ఁפ׮. 2. sampling ߑߨী ٮۄࢲ ࢠ೒݂ػ ؘ੉ఠী bias о Ѧܾࣻ ੓णפ׮. 3. ؀ࠗ࠙੄ పझ௼ীࢲח IDژח OOD੄ ޙ੢੄ ѐࣻо ೠ੿੸ੑפ׮. 
 (਋ܻ ӝמ؀ച ؘ੉ఠࣇب Ӓۧભ ƕƕ)
9. ࢶ೯ োҳ / Previous Work ઁݾ ఫझ౟
10. “A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks”, 2017 ICLR 1. Out-of-distribution Detectionਸ ׮ܘ ୭ୡ੄ ֤ޙ 2. Out-of-distribution Detectionী ؀ೠ ޙઁ ੿੄৬ ಣо ߑߨ, рױೠ ߬੉झۄੋ ١ਸ ઁद 3. Maximum Softmax Probability ч੉ thresholdࠁ׮ ௼ݶ in-distribution, ੘ਵݶ out-of-distribution sample੉ۄҊ ࠙ܨ -> ੌઙ੄ max(softmax(p_i)) == confidence https://hoya012.github.io/blog/anomaly-detection-overview-2/
11. “Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks”, 2018 ICLR 1. ੉޷ ೟ण੉ ՘դ neural networkী যځೠ ୶о ೟ण হ੉ рױೞݶࢲ ബҗ੸ਵ۽ out-of-distribution sampleٜਸ ଺ইյ ࣻ ੓਺. 2. Softmax Temperature scaling : in-distribution sampleҗ out-of-distribution sample੄ softmax scoreܳ ࢲ۽ ؊ ݣয૑ѱ ೞח ৉ೡਸ ೣ. Tчਸ ੜ ࢶ੿ೞח Ѫ੉ ઺ਃೣ (T=1000 ࢎਊ) 3. Input preprocessing: ੌઙ੄ ੉޷૑ Augmentation (Image Perturbation) 4. Baselineਸ ௾ Ѻର۽ ٮجܻݴ Out-of-distribution োҳ੄ ࢿ੢ оמࢿਸ ࠁৈળ ֤ޙ 5. ࠄ ֤ޙীࢲח ੉ োҳীࢲ ઁউೠ ߑߨਸ OoD ؘ੉ఠ হ੉ ࢎਊೡ ࣻ ੓ب۾ ѐࢶೣ https://hoya012.github.io/blog/anomaly-detection-overview-2/ ODIN
12. ੽Ӕ ߑߨ / Approach ઁݾ ఫझ౟
13. - ৈ۞ োҳ [36, 29, 13]ীࢲ যڃ ੑ۱੉ٚ softmax ੄ Ѿҗо ؀୓੸ਵ۽ ֫਷ confidence ܳ ࠁ੉ח Ѿҗо ҙ଴غ঻׮. - ৘ܳ ٜয MNIST ࠙ܨ ݽ؛ী Gaussian Noise ܳ ੑ۱ਵ۽ ઱঻חؘ, 91%੄ probਵ۽ যڃ ௿ېझо ৘ஏغ؊ۄ - ೧׼ োҳٜ਷ ੉۞ೠ Ѿҗо smooth ೠ ૑दೣࣻ৬ ਬࢎೠ softmax੄ ౠࢿ ٸޙ੉ۄҊ ઱੢ೠ׮ - ૊ softmaxח যڃ ೠ classܳ ࠙ݺೞѱ ૑दೞח ౠࢿਸ ыҊ ੓׮. - ٮۄࢲ softmax੄ Ѿҗ۽ uniform ೠ distribution ੉ ইצ spiky (ࡶ઒ೠ) distribution੉ ٜ݅য ૓׮Ҋ ݈੉׮. - ੉ܳ ೧Ѿೞӝ ਤ೧ softmax temperature scaling э਷ ߑߨ੉ ࢎਊغ঻׮. -> ખ؊ uniform ೞѱ غب۾ Over Confidence problem p(y|x) ਋ܻо ೟ण दఅ classifier: ੑ۱ x ী ؀ೠ class probability Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
14. The Decomposed Confidence Score - ೞ૑݅ ࠄ ֤ޙ੄ ੷੗ח softmax੄ ޙઁ ࠁ׮ח ݽ؛੄ ୹۱чী ؊ ޙઁо ੓׮ח Ѫী ୡ੼ਸ ݏ୶Ҋ ҒҒ൤ ࢤп೧ ࠁও׮. - Ӓ җ੿ীࢲ overconfidence о غח ਗੋ੉ softmax о ইצ ݽ؛੄ ୹۱чਸ ࢸ҅ೠ ࠗ࠙ী ੓׮ח Ѫਸ ঌѱ ؽ p(y|x) ਋ܻо ೟ण दఅ classifier: ੑ۱ x ী ؀ೠ class probability ਷ بݫੋী ؀ೠ condition੉ ੹ഃ غয ੓૑ ঋ਺ ૊ ݽٚ ੑ۱੉ x ∈ d in t "In-Distribtuion ী ࣘೞח ؘ੉ఠ"ۄח о੿੉ ನೣغয ੓׮. Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
15. ೞ૑݅ ਋ܻח ੑ۱ਵ۽ In-Distributionҗ Out-of-Distribution ੋ ࢚ടਸ ݽف ׮ܖӝ ٸޙী, domain ী ؀ೠ conditionalೞ ѱ output distribution ੉ ׳ۄઉঠ ೣ. Ӓېࢲ ݺद੸ਵ۽ domain ী ؀ೠ ઑѤࠗ ഛܫਸ ҳೞب۾ ਤ धਸ ߸҃ೣ The Decomposed Confidence Score - ೞ૑݅ ࠄ ֤ޙ੄ ੷੗ח softmax੄ ޙઁ ࠁ׮ח ݽ؛੄ ୹۱чী ؊ ޙઁо ੓׮ח Ѫী ୡ੼ਸ ݏ୶Ҋ ҒҒ൤ ࢤп೧ ࠁও׮. - Ӓ җ੿ীࢲ overconfidence о غח ਗੋ੉ softmax о ইצ ݽ؛੄ ୹۱чਸ ࢸ҅ೠ о੿ী ੓׮ח Ѫਸ ঌѱ ؽ p(y|x) ਋ܻо ೟ण दఅ classifier: ੑ۱ x ী ؀ೠ class probability ਷ بݫੋী ؀ೠ condition੉ ੹ഃ غয ੓૑ ঋ਺ ૊ ݽٚ ੑ۱੉ x ∈ d in t "In-Distribtuion ী ࣘೞח ؘ੉ఠ"ۄח о੿੉ ನೣغয ੓׮. p(y|d in , x) domainਸ Ҋ۰ೞח ೟णػ classifier : ੑ۱ x ী ؀ೠ class probability Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
16. The Decomposed Confidence Score p(y|d in , x) domainਸ Ҋ۰ೞח ೟णػ classifier : ੑ۱ x ী ؀ೠ class probability p(y|d in , x) = p(y, d in |x) p(din |x) ਤ ध਷ ઑѤࠗ ഛܫী ٮۄࢲ ׮਺җ э੉ ੹ѐೡ ࣻ ੓׮. ਋ܻח ੉ ੹ѐػ धਸ ా೧ࢲ ৵ overconfidence о ߊࢤ೮ח૑ ঌ ࣻ ੓׮. Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
17. ઁݾ ఫझ౟ The Decomposed Confidence Score p(y|d in , x) domainਸ Ҋ۰ೞח ೟णػ classifier : ੑ۱ x ী ؀ೠ class probability p(y|d in , x) = p(y, d in |x) p(din |x) ਤ ध਷ ઑѤࠗ ഛܫী ٮۄࢲ ׮਺җ э੉ ੹ѐೡ ࣻ ੓׮. ਋ܻח ੉ ੹ѐػ धਸ ా೧ࢲ ৵ overconfidence о ߊࢤ೮ח૑ ঌ ࣻ ੓׮. ৘ܳ ٜয xо Out-of-Distribution ীࢲ sampling غ঻׮Ҋ ೡ ٸ, p(y, d in |x) ਷ ծਸ чਸ ыਸ Ѫ੐. (৘: 0.09) p(d in |x) ৉द ծਸ чਸ ыਸ Ѫ੐. (৘: 0.1) xо in-distribtution ੌ ഛܫਸ աఋղח y੄ class ഛܫҗ xо in-distribution ੌ ഛܫ੄ Ѿ೤࠙ನ
18. ઁݾ ఫझ౟ The Decomposed Confidence Score p(y|d in , x) domainਸ Ҋ۰ೞח ೟णػ classifier : ੑ۱ x ী ؀ೠ class probability p(y|d in , x) = p(y, d in |x) p(din |x) ਤ ध਷ ઑѤࠗ ഛܫী ٮۄࢲ ׮਺җ э੉ ੹ѐೡ ࣻ ੓׮. ਋ܻח ੉ ੹ѐػ धਸ ా೧ࢲ ৵ overconfidence о ߊࢤ೮ח૑ ঌ ࣻ ੓׮. ৘ܳ ٜয xо Out-of-Distribution ীࢲ sampling غ঻׮Ҋ ೡ ٸ, p(y, d in |x) ਷ ծਸ чਸ ыਸ Ѫ੐. (৘: 0.09) p(d in |x) ৉द ծਸ чਸ ыਸ Ѫ੐. (৘: 0.1) p(y|d in , x) = p(y, d in |x) p(din |x) = 0.09 0.1 = 0.9 xо in-distribtution ੌ ഛܫਸ աఋղח y੄ class ഛܫҗ xо in-distribution ੌ ഛܫ੄ Ѿ೤࠙ನ ૊ xо out-of-distribution ীࢲ ࡳഊ਺ ীب Ѿҗ੸ਵ۽ যڃ class੄ confidence ח ݒ਋ ֫ѱ աৡ׮ח Ѫਸ ঌ ࣻ ੓਺
19. The Decomposed Confidence Score p(y|d in , x) = p(y, d in |x) p(din |x) - ਤ৬ э਷ overconfidence ޙઁܳ ೧Ѿೞח о੢ ૒ҙ੸ੋ ߑߨ਷ 
 y(௿ېझ), d(بݫੋ) ۨ੉࠶ਸ ੉ਊ೧ࢲ ׮਺ ف ݽ؛ਸ supervised-learning ਵ۽ ೟णೞח Ѫ੉׮. - Ӓ۞ݶ classب ৘ஏೡ ࣻ ੓Ҋ, threshold ܳ ੉ਊ೧ࢲ OoD ৉द ࠙߹ೡ ࣻ ੓׮. - ೞ૑݅ ੉ ߑߨ਷ OoD ؘ੉ఠࣇ੉ ೙ਃೠ ߑߨ੉׮. ٸޙী ੉ ޙઁ о੿ী ݏ૑ ঋח׮. - = Out-of-Distribution ীࢲ ࢠ೒݂ ೞݶ ੜ উغҊ ~ ١١ p(y, d in |x) p(d in |x) arg max i (p(y i , d in |x)) p(d in |x) Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
20. - ѾҴ supervised learning হ੉ ׮਺ ف ഛܫ ݽ؛ਸ ೟णೞח Ѫ਷ Ӕࠄ੸ਵ۽ ࠛоמೣ. - ੉۠ ࢚ടীࢲח “self-supervised learning” ژח “unsupervised-learning” ੽Ӕ ߑߨਸ ࢤп೧ ࠅ ࣻ ੓׮. - ਤ৬ э਷ ੽Ӕ ߑߨਸ ࢎਊೞ۰ݶ, ݽ؛ীѱ “ࢎ੹ ૑ध" ژח “о੿”ਸ ࠗৈ೧ ઻ঠ ೣ - ࠄ ֤ޙীࢲח ࠙੗/࠙ݽ ҳઑ੄ धਸ classifierܳ ٜ݅ٸ ਋ܻо ೟णೞҊ੗ ೞח పझ௼੄ ࢎ੹ ੿ࠁ۽ ઱ח ߑधਸ ࢎਊೞ৓׮. - ٍীࢲ ҅ࣘ ׮ܖѷ૑݅ ਤ धҗ э਷ ҳઑח, class probability ੄ confidence ܳ decompose ೡ ࣻ ੓ח מ۱ਸ ࠗৈೠ׮ The Decomposed Confidence Score p(y|d in , x) = p(y, d in |x) p(din |x) p(y, d in |x) p(d in |x) class_prob Decomposed confidence #1 Decomposed confidence #2 Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
21. The Decomposed Confidence Score p(y|d in , x) = p(y, d in |x) p(din |x) Decomposed confidence #1 Decomposed confidence #2 Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
22. The Decomposed Confidence p(y|d in , x) = p(y, d in |x) p(din |x) f i (x) = h i (x) g (x) Cro ssEn tro py(so ftmax(f i (x))) - ؘ੉ఠо In-distribution ੄ ੸਷ ࠙ನܳ ର૑ೞח ҃਋ Multi-class classification ೟ण p(y, d in |x) = h i (x) − > min imize g (x) − > min imize ৵ջೞݶ p(y, d in |x) ܳ ѾҴ ఃਕঠ ೞӝ ٸޙ p(y, d in |x) = h i (x) − > g reato r g (x) − > n o t − min imize ৵ջೞݶ p(y, d in |x) ః਋ח ߑೱਵ۽ ੉޷ h(x)о ೞҊ ੓਺ - ؘ੉ఠо In-distribution ੄ ݆਷ ࠙ನܳ ର૑ೞח ҃਋ p(y, d in |x) ః਋ب۾ ೟ण೧ঠ ೣ - ೟ण: dataо in-distribution ੋ ҃਋ীח ೦࢚ Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
23. The Decomposed Confidence p(y|d in , x) = p(y, d in |x) p(din |x) f i (x) = h i (x) g (x) g_x = sigmoid(BatchNorm(linear(backbone))) h_x_i = linear(backbone) h_x_E = -1 * L2(backbone, w_i) h_x_C = CosineSim(backbone, w_i) h_i(x)ח class ࠙ܨӝ g(x)ח in-distribution ੌ ഛܫ regressor प೷ਸ ా೧ࢲ 3о૑ ݽ؛઺ যڃ Ѫ੉ ઁੌ જ਷૑ ࠺Ү Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
24. The Decomposed Confidence p(y|d in , x) = p(y, d in |x) p(din |x) f i (x) = h i (x) g (x) S DeCo n f (x) = max i h i (x) o r g (x) pred label = arg max i f i (x) o r arg max i h i (x) Out-of-Distribution Score Classification DeConf-I, DeConf-E, and DeConf-C ח h_i(x)ܳ ࢎਊೠ Ѫ যڃ Ѫਸ ࢶఖೡ ૑ח प೷ਸ ా೧ࢲ ঌই ࠁب۾ ೞ੗ Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
25. Dataset ID Dataset - CVHN - CIFAR - 10 - CIFAR - 100 OOD Dataset - TinyImageNet - LSUN - iSUN - Uniform Random Image - Gaussian Random Image Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
26. Ѿҗ / Results ઁݾ ఫझ౟
27. Metric ࢸݺ Area Under - Receiver operating characteristic curve AUROC FPR: Negative ੋؘ Positive ۽ ৘ஏೠ ࠺ਯ / 1- OOD recall TPR: Positive ੋؘ Positive ۽ ৘ஏೠ ࠺ਯ / ID-recall TNR@TPR95 True Negative Rate @ True Positive Rate 95 Positive ੋؘ Positive ۽ ৘ஏೠ ࠺ਯ੉ 95%ੌٸ੄ 
 threshold ীࢲ Negative ੋؘ Negative ۽ ৘ஏೠ ࠺ਯ ૊, ID-recall == 0.95 о غח 
 threshold ীࢲ OOD-recall ਷? Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
28. 4ѐ੄ OoD method ࢿמ ࠺Ү 4о૑ method ݽف OoD ؘ੉ఠܳ ࠁ૑ ঋח о੿ೞী evaluation ೠ Ѿҗ ೟ण ؘ੉ఠ(ID) ೟णؘ੉ఠ x
 OoD ࢿמ ഛੋਊ Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
29. Decomposed Confidence ߑध ࠺Ү Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
30. Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data ৈӝࢲ ഛੋ೧ ࠅ ࣻ ੓ח അ࢚ - (c)৬ (d)ח ׀ীڸѱ ੜ clustering ੉ غয ੓׮ח Ѫ੉׮. - (e)ח ੹୓੸ਵ۽ score о ֫਷ؘ, ੉ח equeation4 ীࢲ ׮ܟ؍ overconfidence അ࢚ ٸޙ੐
31. хࢎ೤פ׮✌ ୶о ૕ޙ ژח ҾӘೠ ੼੉ ੓׮ݶ ঱ઁٚ ইې োۅ୊۽ োۅ ઱ࣁਃ! ӣળࢿ (ML Research Scientist) Email.junseong.kim@scatterlab.co.kr Facebook. @codertimo