Generalized ODIN

Generalized ODIN

A42dd3541cd40296dcd8a5e6b4a01bef?s=128

Scatter Lab Inc.

April 24, 2020
Tweet

Transcript

  1. (FOFSBMJ[FE0%*/ %FUFDUJOH0VUPGEJTUSJCVUJPO*NBHFXJUIPVU -FBSOJOHGSPN0VUPGEJTUSJCVUJPO%BUB ӣળࢿ .BDIJOF-FBSOJOH3FTFBSDI4DJFOUJTU

  2. Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data

    ݾର! 1. ਃড (Abstract) 2. ߓ҃૑ध (Background) :Out-of-Distribution Detection పझ௼ח ޖ঺ੋо? 3. ࢶ೯ োҳ (Previous Work) 4. ޙઁ ੿੄ (Problem Setting) 5. ੽Ӕ ߑߨ (Approach) 6. प೷ (Experiment) 7. Ѿҗ (Results) 8. ࠙ࢳ (Analysis)
  3. ਃড Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution

    Data • ࠄ ֤ޙীࢲח ೟ण ؘ੉ఠ৬ زੌೞ૑ ঋ਷ ࠙ನ੄ ੑ۱੉ ٜযৡ ࢚ടਸ х૑ೞח 
 OoD(Out-of-Distribution) Detection ޙઁܳ ಽҊ੗ ೤פ׮. • ӝઓ੄ ੽Ӕ ߑधٜ਷ “In-Distribution”җ “Out-of-Distribution" ف ؘ੉ఠ ݽف ೟ण ژח ೞ੉ಌ౵ۄ޷ఠ ౚ׬ী ࢎਊೞ৓णפ׮. • ࠄ ֤ޙীࢲח OoD ؘ੉ఠܳ ੌ੺ ࢎਊೞ૑ ঋח ࢚ടীࢲ ੉ ޙઁܳ ೧ѾೞҊ੗ ೤פ׮. • ೧Ѿ ߑߨਵ۽ࢲ, ഛܫ੸ ੽Ӕਸ ߄ఔਵ۽ ೞח “Decomposed Confidence” ۄח ߑߨਸ ઁউ೤פ׮. • प೷ਸ ా೧ OoD ؘ੉ఠܳ ࢎਊೞৈ ೟णೠ baseline ݽ؛ٜ ࠁ׮ ਋ࣻೠ ࢿמਸ Ѣل ࣻ ੓঻਺ਸ ࠁৈસפ׮.
  4. ߓ҃ ૑ध / Background ઁݾ ఫझ౟

  5. Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data

    Out-of-Distribution(OoD) Detection ੉ۆ? - ೟णػ ݽ؛ਸ प ࢲ࠺झী ߓನೞݶ, ೟णदী ࠁ૑ ޅ೮؍ ੑ۱ٜ(=Out-of-Distribution)ਸ ࠁѱ ؽ - ૑ӘԈ ࠁ૑ ޅೠ ੑ۱ী ؀೧ࢲ ML ݽ؛ٜ਷ ҭ੢൤ ੜޅػ ز੘ ژח ৘ஏਸ ೣ - ೞ૑݅ ݽ؛਷ ੉ѱ ੜޅػ ੑ۱ੋ૑ ઁ؀۽ ػ ੑ۱ੋ૑ ౸ױೡ ࣻ ੓ח מ۱੉ হ਺ - ৘ܳ ٜযࢲ, ਺धਸ ҳ࠙ೞח image classification model ੉ ੓׮Ҋ ೮ਸ ٸ, 
 Airbnb ߑ ࢎ૓੉ ٜযয়ݶ যڌѱ ؼө? ಃா੉௼? ࢎҗ? - ਋ܽ ੉۠ Ѿҗٜ੉ ৘ஏغӡ ߄ۄ૑ ঋҊ, “৘ஏೡ ࣻ হ਺”ਸ ߜӡ ਗೠ׮
  6. Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data

  7. Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data

    Out-of-Distribution(OoD) Detection ੉ۆ? - ਤীࢲ ࢸݺೠ ৘दܳ ࠁ׮ ҳ୓੸ੋ ۨ߰ীࢲ ޙઁܳ ੿੄ೠ׮ݶ - In-Distribution(Nѐ੄ class) ؘ੉ఠ۽ ೟णػ Multi-Class Classification ݽ؛੉ ੓ਸ ٸ - పझ౟ ؘ੉ఠࣇ੄ যڃ ੑ۱ xо In-Distribution ী ࣘ೧੓׮ݶ x੄ ੿ഛೠ classܳ ৘ஏೞӝ - xо In-Distribution ী ࣘ೧ ੓ח ׮ח Ѫ਷, ID ղ੄ যڃ classী ೧׼ ػ׮ח Ѫ - పझ౟ ؘ੉ఠࣇ੄ যڃ ੑ۱ x*о In-Distribution ী ࣘ೧੓૑ ঋ׮ݶ, OoD ੑ۱੐ਸ х૑ೞח Ѫ - x*о In-Distribution ী ࣘ೧੓૑ ঋ׮ח Ѫ਷, ID ղ੄ যڃ classীب ೧׼غ૑ ঋח ׮ח Ѫ
  8. Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data

    ૕ޙ੓णפ׮ ૕ޙ! - Q. ID, OOD ܳ ҳ࠙ೞח binary classification model ਸ ٜ݅ݶ غחѢ ইצоਃ? - A. উఋӰѱب Ӓ ߑߨীח ݻо૑ ޙઁ੼੉ ઓ੤೤פ׮. 1. classification ೟णী ࢎਊೡ OOD ؘ੉ఠܳ ID ؘ੉ఠ ѐࣻ ݅ఀ OOD ҕрীࢲ sampling ೞݶ OOD ҕр ੹୓ܳ ׮ ಴അೞ૑ ޅೞח ޙઁо ੓णפ׮. ౠ൤ OOD ҕр੉ ௼ݶ ௿ ࣻ۾ ੉ ޙઁח ؊ ఁפ׮. 2. sampling ߑߨী ٮۄࢲ ࢠ೒݂ػ ؘ੉ఠী bias о Ѧܾࣻ ੓णפ׮. 3. ؀ࠗ࠙੄ పझ௼ীࢲח IDژח OOD੄ ޙ੢੄ ѐࣻо ೠ੿੸ੑפ׮. 
 (਋ܻ ӝמ؀ച ؘ੉ఠࣇب Ӓۧભ ƕƕ)
  9. ࢶ೯ োҳ / Previous Work ઁݾ ఫझ౟

  10. “A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural

    Networks”, 2017 ICLR 1. Out-of-distribution Detectionਸ ׮ܘ ୭ୡ੄ ֤ޙ 2. Out-of-distribution Detectionী ؀ೠ ޙઁ ੿੄৬ ಣо ߑߨ, рױೠ ߬੉झۄੋ ١ਸ ઁद 3. Maximum Softmax Probability ч੉ thresholdࠁ׮ ௼ݶ in-distribution, ੘ਵݶ out-of-distribution sample੉ۄҊ ࠙ܨ -> ੌઙ੄ max(softmax(p_i)) == confidence https://hoya012.github.io/blog/anomaly-detection-overview-2/
  11. “Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks”,

    2018 ICLR 1. ੉޷ ೟ण੉ ՘դ neural networkী যځೠ ୶о ೟ण হ੉ рױೞݶࢲ ബҗ੸ਵ۽ out-of-distribution sampleٜਸ ଺ইյ ࣻ ੓਺. 2. Softmax Temperature scaling : in-distribution sampleҗ out-of-distribution sample੄ softmax scoreܳ ࢲ۽ ؊ ݣয૑ѱ ೞח ৉ೡਸ ೣ. Tчਸ ੜ ࢶ੿ೞח Ѫ੉ ઺ਃೣ (T=1000 ࢎਊ) 3. Input preprocessing: ੌઙ੄ ੉޷૑ Augmentation (Image Perturbation) 4. Baselineਸ ௾ Ѻର۽ ٮجܻݴ Out-of-distribution োҳ੄ ࢿ੢ оמࢿਸ ࠁৈળ ֤ޙ 5. ࠄ ֤ޙীࢲח ੉ োҳীࢲ ઁউೠ ߑߨਸ OoD ؘ੉ఠ হ੉ ࢎਊೡ ࣻ ੓ب۾ ѐࢶೣ https://hoya012.github.io/blog/anomaly-detection-overview-2/ ODIN
  12. ੽Ӕ ߑߨ / Approach ઁݾ ఫझ౟

  13. - ৈ۞ োҳ [36, 29, 13]ীࢲ যڃ ੑ۱੉ٚ softmax ੄

    Ѿҗо ؀୓੸ਵ۽ ֫਷ confidence ܳ ࠁ੉ח Ѿҗо ҙ଴غ঻׮. - ৘ܳ ٜয MNIST ࠙ܨ ݽ؛ী Gaussian Noise ܳ ੑ۱ਵ۽ ઱঻חؘ, 91%੄ probਵ۽ যڃ ௿ېझо ৘ஏغ؊ۄ - ೧׼ োҳٜ਷ ੉۞ೠ Ѿҗо smooth ೠ ૑दೣࣻ৬ ਬࢎೠ softmax੄ ౠࢿ ٸޙ੉ۄҊ ઱੢ೠ׮ - ૊ softmaxח যڃ ೠ classܳ ࠙ݺೞѱ ૑दೞח ౠࢿਸ ыҊ ੓׮. - ٮۄࢲ softmax੄ Ѿҗ۽ uniform ೠ distribution ੉ ইצ spiky (ࡶ઒ೠ) distribution੉ ٜ݅য ૓׮Ҋ ݈੉׮. - ੉ܳ ೧Ѿೞӝ ਤ೧ softmax temperature scaling э਷ ߑߨ੉ ࢎਊغ঻׮. -> ખ؊ uniform ೞѱ غب۾ Over Confidence problem p(y|x) ਋ܻо ೟ण दఅ classifier: ੑ۱ x ী ؀ೠ class probability Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
  14. The Decomposed Confidence Score - ೞ૑݅ ࠄ ֤ޙ੄ ੷੗ח softmax੄

    ޙઁ ࠁ׮ח ݽ؛੄ ୹۱чী ؊ ޙઁо ੓׮ח Ѫী ୡ੼ਸ ݏ୶Ҋ ҒҒ൤ ࢤп೧ ࠁও׮. - Ӓ җ੿ীࢲ overconfidence о غח ਗੋ੉ softmax о ইצ ݽ؛੄ ୹۱чਸ ࢸ҅ೠ ࠗ࠙ী ੓׮ח Ѫਸ ঌѱ ؽ p(y|x) ਋ܻо ೟ण दఅ classifier: ੑ۱ x ী ؀ೠ class probability ਷ بݫੋী ؀ೠ condition੉ ੹ഃ غয ੓૑ ঋ਺ ૊ ݽٚ ੑ۱੉ x ∈ d in t "In-Distribtuion ী ࣘೞח ؘ੉ఠ"ۄח о੿੉ ನೣغয ੓׮. Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
  15. ೞ૑݅ ਋ܻח ੑ۱ਵ۽ In-Distributionҗ Out-of-Distribution ੋ ࢚ടਸ ݽف ׮ܖӝ ٸޙী,

    domain ী ؀ೠ conditionalೞ ѱ output distribution ੉ ׳ۄઉঠ ೣ. Ӓېࢲ ݺद੸ਵ۽ domain ী ؀ೠ ઑѤࠗ ഛܫਸ ҳೞب۾ ਤ धਸ ߸҃ೣ The Decomposed Confidence Score - ೞ૑݅ ࠄ ֤ޙ੄ ੷੗ח softmax੄ ޙઁ ࠁ׮ח ݽ؛੄ ୹۱чী ؊ ޙઁо ੓׮ח Ѫী ୡ੼ਸ ݏ୶Ҋ ҒҒ൤ ࢤп೧ ࠁও׮. - Ӓ җ੿ীࢲ overconfidence о غח ਗੋ੉ softmax о ইצ ݽ؛੄ ୹۱чਸ ࢸ҅ೠ о੿ী ੓׮ח Ѫਸ ঌѱ ؽ p(y|x) ਋ܻо ೟ण दఅ classifier: ੑ۱ x ী ؀ೠ class probability ਷ بݫੋী ؀ೠ condition੉ ੹ഃ غয ੓૑ ঋ਺ ૊ ݽٚ ੑ۱੉ x ∈ d in t "In-Distribtuion ী ࣘೞח ؘ੉ఠ"ۄח о੿੉ ನೣغয ੓׮. p(y|d in , x) domainਸ Ҋ۰ೞח ೟णػ classifier : ੑ۱ x ী ؀ೠ class probability Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
  16. The Decomposed Confidence Score p(y|d in , x) domainਸ Ҋ۰ೞח

    ೟णػ classifier : ੑ۱ x ী ؀ೠ class probability p(y|d in , x) = p(y, d in |x) p(din |x) ਤ ध਷ ઑѤࠗ ഛܫী ٮۄࢲ ׮਺җ э੉ ੹ѐೡ ࣻ ੓׮. ਋ܻח ੉ ੹ѐػ धਸ ా೧ࢲ ৵ overconfidence о ߊࢤ೮ח૑ ঌ ࣻ ੓׮. Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
  17. ઁݾ ఫझ౟ The Decomposed Confidence Score p(y|d in , x)

    domainਸ Ҋ۰ೞח ೟णػ classifier : ੑ۱ x ী ؀ೠ class probability p(y|d in , x) = p(y, d in |x) p(din |x) ਤ ध਷ ઑѤࠗ ഛܫী ٮۄࢲ ׮਺җ э੉ ੹ѐೡ ࣻ ੓׮. ਋ܻח ੉ ੹ѐػ धਸ ా೧ࢲ ৵ overconfidence о ߊࢤ೮ח૑ ঌ ࣻ ੓׮. ৘ܳ ٜয xо Out-of-Distribution ীࢲ sampling غ঻׮Ҋ ೡ ٸ, p(y, d in |x) ਷ ծਸ чਸ ыਸ Ѫ੐. (৘: 0.09) p(d in |x) ৉द ծਸ чਸ ыਸ Ѫ੐. (৘: 0.1) xо in-distribtution ੌ ഛܫਸ աఋղח y੄ class ഛܫҗ xо in-distribution ੌ ഛܫ੄ Ѿ೤࠙ನ
  18. ઁݾ ఫझ౟ The Decomposed Confidence Score p(y|d in , x)

    domainਸ Ҋ۰ೞח ೟णػ classifier : ੑ۱ x ী ؀ೠ class probability p(y|d in , x) = p(y, d in |x) p(din |x) ਤ ध਷ ઑѤࠗ ഛܫী ٮۄࢲ ׮਺җ э੉ ੹ѐೡ ࣻ ੓׮. ਋ܻח ੉ ੹ѐػ धਸ ా೧ࢲ ৵ overconfidence о ߊࢤ೮ח૑ ঌ ࣻ ੓׮. ৘ܳ ٜয xо Out-of-Distribution ীࢲ sampling غ঻׮Ҋ ೡ ٸ, p(y, d in |x) ਷ ծਸ чਸ ыਸ Ѫ੐. (৘: 0.09) p(d in |x) ৉द ծਸ чਸ ыਸ Ѫ੐. (৘: 0.1) p(y|d in , x) = p(y, d in |x) p(din |x) = 0.09 0.1 = 0.9 xо in-distribtution ੌ ഛܫਸ աఋղח y੄ class ഛܫҗ xо in-distribution ੌ ഛܫ੄ Ѿ೤࠙ನ ૊ xо out-of-distribution ীࢲ ࡳഊ਺ ীب Ѿҗ੸ਵ۽ যڃ class੄ confidence ח ݒ਋ ֫ѱ աৡ׮ח Ѫਸ ঌ ࣻ ੓਺
  19. The Decomposed Confidence Score p(y|d in , x) = p(y,

    d in |x) p(din |x) - ਤ৬ э਷ overconfidence ޙઁܳ ೧Ѿೞח о੢ ૒ҙ੸ੋ ߑߨ਷ 
 y(௿ېझ), d(بݫੋ) ۨ੉࠶ਸ ੉ਊ೧ࢲ ׮਺ ف ݽ؛ਸ supervised-learning ਵ۽ ೟णೞח Ѫ੉׮. - Ӓ۞ݶ classب ৘ஏೡ ࣻ ੓Ҋ, threshold ܳ ੉ਊ೧ࢲ OoD ৉द ࠙߹ೡ ࣻ ੓׮. - ೞ૑݅ ੉ ߑߨ਷ OoD ؘ੉ఠࣇ੉ ೙ਃೠ ߑߨ੉׮. ٸޙী ੉ ޙઁ о੿ী ݏ૑ ঋח׮. - = Out-of-Distribution ীࢲ ࢠ೒݂ ೞݶ ੜ উغҊ ~ ١١ p(y, d in |x) p(d in |x) arg max i (p(y i , d in |x)) p(d in |x) Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
  20. - ѾҴ supervised learning হ੉ ׮਺ ف ഛܫ ݽ؛ਸ ೟णೞח

    Ѫ਷ Ӕࠄ੸ਵ۽ ࠛоמೣ. - ੉۠ ࢚ടীࢲח “self-supervised learning” ژח “unsupervised-learning” ੽Ӕ ߑߨਸ ࢤп೧ ࠅ ࣻ ੓׮. - ਤ৬ э਷ ੽Ӕ ߑߨਸ ࢎਊೞ۰ݶ, ݽ؛ীѱ “ࢎ੹ ૑ध" ژח “о੿”ਸ ࠗৈ೧ ઻ঠ ೣ - ࠄ ֤ޙীࢲח ࠙੗/࠙ݽ ҳઑ੄ धਸ classifierܳ ٜ݅ٸ ਋ܻо ೟णೞҊ੗ ೞח పझ௼੄ ࢎ੹ ੿ࠁ۽ ઱ח ߑधਸ ࢎਊೞ৓׮. - ٍীࢲ ҅ࣘ ׮ܖѷ૑݅ ਤ धҗ э਷ ҳઑח, class probability ੄ confidence ܳ decompose ೡ ࣻ ੓ח מ۱ਸ ࠗৈೠ׮ The Decomposed Confidence Score p(y|d in , x) = p(y, d in |x) p(din |x) p(y, d in |x) p(d in |x) class_prob Decomposed confidence #1 Decomposed confidence #2 Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
  21. The Decomposed Confidence Score p(y|d in , x) = p(y,

    d in |x) p(din |x) Decomposed confidence #1 Decomposed confidence #2 Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
  22. The Decomposed Confidence p(y|d in , x) = p(y, d

    in |x) p(din |x) f i (x) = h i (x) g (x) Cro ssEn tro py(so ftmax(f i (x))) - ؘ੉ఠо In-distribution ੄ ੸਷ ࠙ನܳ ର૑ೞח ҃਋ Multi-class classification ೟ण p(y, d in |x) = h i (x) − > min imize g (x) − > min imize ৵ջೞݶ p(y, d in |x) ܳ ѾҴ ఃਕঠ ೞӝ ٸޙ p(y, d in |x) = h i (x) − > g reato r g (x) − > n o t − min imize ৵ջೞݶ p(y, d in |x) ః਋ח ߑೱਵ۽ ੉޷ h(x)о ೞҊ ੓਺ - ؘ੉ఠо In-distribution ੄ ݆਷ ࠙ನܳ ର૑ೞח ҃਋ p(y, d in |x) ః਋ب۾ ೟ण೧ঠ ೣ - ೟ण: dataо in-distribution ੋ ҃਋ীח ೦࢚ Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
  23. The Decomposed Confidence p(y|d in , x) = p(y, d

    in |x) p(din |x) f i (x) = h i (x) g (x) g_x = sigmoid(BatchNorm(linear(backbone))) h_x_i = linear(backbone) h_x_E = -1 * L2(backbone, w_i) h_x_C = CosineSim(backbone, w_i) h_i(x)ח class ࠙ܨӝ g(x)ח in-distribution ੌ ഛܫ regressor प೷ਸ ా೧ࢲ 3о૑ ݽ؛઺ যڃ Ѫ੉ ઁੌ જ਷૑ ࠺Ү Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
  24. The Decomposed Confidence p(y|d in , x) = p(y, d

    in |x) p(din |x) f i (x) = h i (x) g (x) S DeCo n f (x) = max i h i (x) o r g (x) pred label = arg max i f i (x) o r arg max i h i (x) Out-of-Distribution Score Classification DeConf-I, DeConf-E, and DeConf-C ח h_i(x)ܳ ࢎਊೠ Ѫ যڃ Ѫਸ ࢶఖೡ ૑ח प೷ਸ ా೧ࢲ ঌই ࠁب۾ ೞ੗ Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
  25. Dataset ID Dataset - CVHN - CIFAR - 10 -

    CIFAR - 100 OOD Dataset - TinyImageNet - LSUN - iSUN - Uniform Random Image - Gaussian Random Image Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
  26. Ѿҗ / Results ઁݾ ఫझ౟

  27. Metric ࢸݺ Area Under - Receiver operating characteristic curve AUROC

    FPR: Negative ੋؘ Positive ۽ ৘ஏೠ ࠺ਯ / 1- OOD recall TPR: Positive ੋؘ Positive ۽ ৘ஏೠ ࠺ਯ / ID-recall TNR@TPR95 True Negative Rate @ True Positive Rate 95 Positive ੋؘ Positive ۽ ৘ஏೠ ࠺ਯ੉ 95%ੌٸ੄ 
 threshold ীࢲ Negative ੋؘ Negative ۽ ৘ஏೠ ࠺ਯ ૊, ID-recall == 0.95 о غח 
 threshold ীࢲ OOD-recall ਷? Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
  28. 4ѐ੄ OoD method ࢿמ ࠺Ү 4о૑ method ݽف OoD ؘ੉ఠܳ

    ࠁ૑ ঋח о੿ೞী evaluation ೠ Ѿҗ ೟ण ؘ੉ఠ(ID) ೟णؘ੉ఠ x
 OoD ࢿמ ഛੋਊ Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data
  29. Decomposed Confidence ߑध ࠺Ү Generalized ODIN: Detecting Out-of-distribution Image without

    Learning from Out-of-distribution Data
  30. Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data

    ৈӝࢲ ഛੋ೧ ࠅ ࣻ ੓ח അ࢚ - (c)৬ (d)ח ׀ীڸѱ ੜ clustering ੉ غয ੓׮ח Ѫ੉׮. - (e)ח ੹୓੸ਵ۽ score о ֫਷ؘ, ੉ח equeation4 ীࢲ ׮ܟ؍ overconfidence അ࢚ ٸޙ੐
  31. хࢎ೤פ׮✌ ୶о ૕ޙ ژח ҾӘೠ ੼੉ ੓׮ݶ ঱ઁٚ ইې োۅ୊۽

    োۅ ઱ࣁਃ! ӣળࢿ (ML Research Scientist) Email.junseong.kim@scatterlab.co.kr Facebook. @codertimo