目的関数と評価指標 / Objective Function and Evaluation Metrics

目的関数と評価指標 / Objective Function and Evaluation Metrics

This presentation explains goodness of modeling in machine learning.
After that, show difference between Objective Function and Evaluation Metrics.

5346146aca9add2e6c02b01bf4c0df2e?s=128

jeey

May 15, 2020
Tweet

Transcript

  1. ໨తؔ਺ͱධՁࢦඪ OBJECTIVE FUNCTION AND EVALUATION METRICS

  2. ྑ͍Ϟσϧͬͯ ͳΜͰ͔͢ʁ

  3. ͏͓͓͓͓͓͓͓ʂʂʂʂʂ ਫ਼౓ʂ·͞ʹ̔̕ɽ̕ˋʂ ͨ͗ΔʂԶͷମʂʂ ΄ͱ͹͠Εʂ3.4&ʂʂʂʂ AIΤϯδχΞ

  4. それって現場で ちゃんと使えるの? Ϗδωε୲౰ऀ

  5. モデルの”良さ”は ⼈・場⾯によって 異なる

  6. Ϟσϧͷ ྑ͞ ͱ͸

  7. σʔλ෼ੳͷϑΣʔζʹΑͬͯҟͳΔ ”ྑ͞” ࣮૷ ▸ ໨తؔ਺ɿϞσϧͷֶशʹ͓͍ͯ࠷దԽ͞ΕΔؔ਺ɺ༧ଌ஋ͱ࣮ ଌ஋͕ဃ཭͠ͳ͍ͷ͕”ྑ͞”ʢRMSE΍LoglossͳͲʣ ▸ ධՁࢦඪɿϞσϧ΍༧ଌ஋ͷੑೳͷྑ͠ѱ͠ΛଌΔࢦඪɺ෼ੳ՝ ୊΁ͷޮՌͷߴ͕͞”ྑ͞”ʢRMSE΍AccuracyɺAUCͳͲʣ ▸

    KPIɿϏδωε্Ͱͷ՝୊΁ͷޮՌͷߴ͕͞”ྑ͞” ▸ ROIɿಋೖඅ༻ʹର͢ΔϞσϧར༻ޮՌͷߴ͕͞”ྑ͞” Ϗδωε
  8. ࣮૷Ͱ͸Ͳ͜·Ͱҙࣝ͢΂͖ʁ ▸ Ϟσϧ͸Ϗδωεͷݱ৔Ͱ࢖ΘΕΔ΋ͷͰ͋ΔͨΊɺROI΍KPIͱ͍ͬͨϏδωε্ ͷࢦඪΛϞσϧͷ࠷େԽࢦඪͱ͢Δͷ͕ཧ૝త ▸ ͔͠͠ɺͦΕ͸ݱ࣮తͰ͸ͳ͍ ▸ ͳͥͳΒɺϏδωε্ͷࢦඪ͸ϞσϧΛར༻ͯ͠ҙࢥܾఆͨ݁͠ՌͰ͋Δ͠ɺϞσ ϧͷ݁ՌΛ୭͔Ͳ͏ҙࢥܾఆͯ͠Ϗδωε্ͷ݁Ռ͕΋ͨΒ͞Ε͔ͨͱ͍ͬͨσʔ λ͸ଘࡏ͠ͳ͍

    ▸ ͕ͨͬͯ͠ɺϞσϧߏஙͷϑΣʔζͰ͸ɺҙࢥܾఆͷ໾ʹཱͭ݁ՌΛग़͢ͱ͍͏ϙ Πϯτʹয఺ΛߜͬͯϞσϧͷྑ͞Λఆٛͨ͠΄͏͕ྑ͍ ▸ ͭ·ΓɺϞσϧߏஙͷࡍʹ͸ɺධՁࢦඪΛҙࣝ͢Δͷ͕ݱ࣮త
  9. ໨తؔ਺ͱ ධՁࢦඪͷ ҧ͍

  10. ໨తؔ਺ͱධՁࢦඪͷҧ͍ લड़ͷྑ͞ͷϨϕϧͷҧ͍ʹՃ͑ͯɺଟ͘ͷػցֶशϞσϧʹ͓͍ͯ͸ɺ ໨తؔ਺͸ඍ෼ՄೳͰͳ͚Ε͹ͳΒͳ͍ͷʹର͠ɺධՁࢦඪʹ͸੍໿͕ͳ͍ ▸ Q. ͳͥ໨తؔ਺͸ඍ෼ՄೳͰͳ͚Ε͹ͳΒͳ͍͔ ▸ A. ໨తؔ਺Λ࠷దԽ͢Δ͜ͱ͕ϞσϧΛ࡞Δ͜ͱͷҰ෦Ͱ͋Δ͔Β ͦͯ͠ɺ࠷దԽͱ͸ɺ͋Δ஋͕͍͍ײ͡ʹͳΔͱ͜ΖΛ୳͢ख๏ͷ͜ͱΛࢦ͔͢Β

    ͦͯͦͯ͠͠ɺ͍͍͔Μ͡ʹͳΔͱ͜ΖΛ୳͢ख๏Ͱ͸ɺۃ஋΍มԽ཰ͷܭࢉ͕ඞཁͱͳΔ
  11. ࠷దԽख๏ͱඍ෼ ▸ ࠷খೋ৐๏ શ࣮ଌ஋ʹ͓͍ͯ༧ଌ஋ͱͷ࢒ࠩͷ૯࿨͕࠷খʹͳΔɺϞσϧࣜͷύϥϝʔλΛٻΊΔ ʹޡࠩؔ਺ͷۃখ஋Λ୳͢ͱ͍͏λεΫʹͳΔ ʹ֤ύϥϝʔλʹ͍ͭͯภඍ෼͠ɺ࿈ཱํఔࣜΛ࡞ͬͯղ͘ʢਖ਼نํఔࣜΛղ͘ʣ ʹඍ෼͠ͳ͚Ε͹ͳΒͳ͍ ▸ ࠷໬๏ શ࣮ଌ஋ʹ͓͍ͯͦΕΒͷ஋ΛऔΔ֬཰ͷ૬৐ʢ໬౓ʣ͕࠷େͱͳΔɺϞσϧࣜͷύϥϝʔλΛٻΊΔ

    ʹ໬౓ؔ਺ͷۃେ஋Λ୳͢ͱ͍͏λεΫʹͳΔ ʹ֤ύϥϝʔλʹ͍ͭͯภඍ෼͠ɺ࿈ཱํఔࣜΛ࡞ͬͯղ͘ ʹඍ෼͠ͳ͚Ε͹ͳΒͳ͍ ▸ ͦͷଞɺޯ഑߱Լ๏ɺ४χϡʔτϯ๏ͳͲ΋ಉ͡ ※ඍ෼Λ༻͍ͳ͍࠷దԽख๏΋͋Δ
  12. ධՁࢦඪΛ ࠷దԽ͢Δࡍͷ Ξϓϩʔν

  13. ධՁࢦඪΛ࠷దԽ͢ΔࡍͷΞϓϩʔν 1. ୯ʹਖ਼͘͠ϞσϦϯάΛߦ͏ ‣ ໨తؔ਺ͱධՁࢦඪʹಉ͡΋ͷΛࢦఆͰ͖Δ࣌ 2. ֶशσʔλͷલॲཧΛͯ͠ɺҟͳΔධՁࢦඪΛ࠷దԽ͢Δ ‣ ධՁࢦඪ͕RMSLEͷͱ͖ɺ໨తؔ਺ʹRMSEΛ࢖༻͠ɺ݁ՌΛࢦ਺ม׵͢ΔͳͲ 3.

    ҟͳΔධՁࢦඪͷ࠷దԽΛߦ͍ɺޙॲཧΛߦ͏ ‣ ධՁࢦඪͷੑ࣭ʹैͬͯσʔλΛॲཧͨ͠Γɺᮢ஋Λ࠷దԽ͢ΔͳͲ 4. ΧελϜ໨తؔ਺Λ࢖༻͢Δ ‣ ධՁࢦඪʹ͍ۙ໨తؔ਺Λ࢖༻͢Δ 5. ҟͳΔධՁࢦඪΛ࠷దԽ͠ɺΞʔϦʔετοϐϯάΛߦ͏
  14. ۩ମྫᶃ 3. F1-score͕ධՁࢦඪͷ৔߹ͷɺᮢ஋ͷ࠷దԽ ▸ F1-score͸precisionͱrecallͷௐ࿨ฏۉͷͨΊɺਖ਼ྫෛྫͷෆۉߧσʔλͷ৔߹͸ɺ୯७ʹ0.5Λ ᮢ஋ͱ͠ͳ͍΄͏͕ྑ͍৔߹͕͋Δ ▸ ͜ͷ৔߹͸ɺᮢ஋Λ࠷దԽΞϧΰϦζϜΛ༻͍ͯ࠷దԽ͢Δඞཁ͕͋Δ

  15. ۩ମྫᶄ 4. MAE͕ධՁࢦඪͷ৔߹ͷɺ໨తؔ਺ͷඍ෼Մೳͳۙࣅؔ਺ͷར༻ ▸ MAE͸ඍ෼ෆՄೳͰ͋ΔͨΊɺ໨తؔ਺ʹͦͷ··ར༻͢Δ͜ͱ͸Ͱ͖ͳ͍ ▸ ͜ͷ৔߹ɺඍ෼ՄೳͳMAEͷۙࣅؔ਺Λ༻͍ΔࣄʹΑͬͯɺධՁࢦඪʹରͯ͠ΑΓ࠷దԽͰ͖Δ Մೳੑ͕͋Δ

  16. ࢀߟจݙ ▸ ໳࿬େีɼࡕాོ࢘ɼอࡔܡ༎ɼฏদ༤࢘ ʢ̎̌̍̕ʣ, KaggleͰউͭσʔλ෼ ੳͷٕज़, ٕज़ධ࿦ࣾ, ୈೋষ λεΫͱධՁࢦඪ ▸

    Datarobot blog Ϟσϧ࠷దԽࢦඪ-ධՁࢦඪͷબͼํ ▸ https://www.datarobot.com/jp/blog/Ϟσϧ࠷దԽࢦඪ-ධՁࢦඪͷબͼํ/