Upgrade to Pro — share decks privately, control downloads, hide ads and more …

About Missing Values

465a530291aedb53fff5a6333e3dedaa?s=47 bk
August 09, 2019

About Missing Values

465a530291aedb53fff5a6333e3dedaa?s=128

bk

August 09, 2019
Tweet

More Decks by bk

Other Decks in Science

Transcript

  1. ܽଛ஋ॲཧʹ͍ͭͯ dࣦΘΕͨ஋ΛٻΊͯd

  2. ໨࣍ 1. ܽଛ஋ͱ͸ 2. ܽଛআڈ 3. ܽଛλΠϓ 4. ୯Ұ୅ೖ๏ 5.

    ଟॏ୅ೖ๏ 6. ࠓճ঺հ͠ͳ͔ͬͨ΋ͷ 7. ࢀߟจݙ
  3. ܽଛ஋ͱ͸ Missing Valueʢܽଛ஋ɺܽଌ஋ʣ In statistics, missing data, or missing values,

    occur when no data value is stored for the variable in an observation. Missing data are a common occurrence and can have a significant effect on the conclusions that can be drawn from the data. ʢhttps://en.wikipedia.org/wiki/Missing_dataʣ
  4. ܽଛ͕͋Δͱ Կ͕ࠔΔͷ͔ʁ ܽଛ஋ͱ͸

  5. ܭࢉग़དྷͳ͍ Q,10, 20, ܽଛ஋ͷฏۉ͸ʁ ܽଛ஋ͱ͸ > mean(c(10, 20, NA)) [1]

    NA
  6. Ͳ͏͢Δͷ͔ʁ ܽଛ஋ͱ͸

  7. 1. ফ͢ 2. ຒΊΔ ܽଛ஋ͱ͸

  8. 1. ফ͢ 2. ຒΊΔ 1. ফ͢ ܽଛ஋ͱ͸

  9. ܽଛͷ͋ΔϨίʔυΛফ͢ ମॏ ੑผ 50 ঁ 70 உ 55 NA NA

    உ ܽଛ஋আڈ
  10. ମॏ ੑผ 50 ঁ 70 உ 55 NA NA உ

    ମॏ ੑผ 50 ঁ 70 உ ܽଛͷ͋ΔϨίʔυΛফ͢ ܽଛ஋আڈ
  11. ମॏ ੑผ 50 ঁ 70 உ 55 NA NA உ

    ମॏ ੑผ 50 ঁ 70 உ ϦετϫΠζ๏ ܽଛͷ͋ΔϨίʔυΛফ͢ ܽଛ஋আڈ
  12. ফ͍͍ͯ͠ͷʁ 1. ফͯ͠΋ྑ͍৔߹ 2. ফͯ͠͸ବ໨ͳ৔߹ ܽଛ஋আڈ

  13. ׬શʹϥϯμϜʹܽଛ ܽଛύλʔϯ͸ੑผ ΍ମॏͱແؔ܎ ܽଛ஋আڈ

  14. ׬શʹϥϯμϜʹܽଛ ܽଛύλʔϯ͸ੑผ ΍ମॏͱແؔ܎ ফͯ͠΋໰୊͸ͳ͍ ͨͩ͠ɺޮ཰͸Լ͕Δ ܽଛ஋আڈ

  15. ؍ଌม਺ʹґଘͯܽ͠ଛ ঁੑͷํ͕ଟܽ͘ଛ ܽଛ஋আڈ

  16. ؍ଌม਺ʹґଘͯܽ͠ଛ ফ͢ͱঁੑͷσʔλͷΈݮগ ภΔ ঁੑͷํ͕ଟܽ͘ଛ ܽଛ஋আڈ

  17. ܽଛม਺ࣗମʹґଘͯܽ͠ଛ ମॏͷॏ͍ํ͕ଟ͘ ܽଛ ܽଛ஋আڈ

  18. ܽଛม਺ࣗମʹґଘͯܽ͠ଛ ফ͢ͱମॏͷߴ͍σʔλ͕ݮগ ภΔ ମॏͷॏ͍ํ͕ଟ͘ ܽଛ ܽଛ஋আڈ

  19. ‣ ׬શʹϥϯμϜʹܽଛˠMCAR ʢMissing Completely At Randomʣ ‣ ؍ଌม਺ʹґଘͯܽ͠ଛˠMAR ʢMissing At

    Randomʣ ‣ ܽଛม਺ࣗମʹґଘͯܽ͠ଛˠNMAR ʢNot Missing At Randomʣ ܽଛλΠϓ
  20. ‣ ׬શʹϥϯμϜʹܽଛˠMCAR ʢMissing Completely At Randomʣ ‣ ؍ଌม਺ʹґଘͯܽ͠ଛˠMAR ʢMissing At

    Randomʣ ‣ ܽଛม਺ࣗମʹґଘͯܽ͠ଛˠNMAR ʢNot Missing At Randomʣ ܽଛλΠϓ
  21. ‣ ׬શʹϥϯμϜʹܽଛˠMCAR ʢMissing Completely At Randomʣ ‣ ؍ଌม਺ʹґଘͯܽ͠ଛˠMAR ʢMissing At

    Randomʣ ‣ ܽଛม਺ࣗମʹґଘͯܽ͠ଛˠNMAR ʢNot Missing At Randomʣ ফͯ͠΋ྑ͍ ܽଛλΠϓ
  22. ‣ ׬શʹϥϯμϜʹܽଛˠMCAR ʢMissing Completely At Randomʣ ‣ ؍ଌม਺ʹґଘͯܽ͠ଛˠMAR ʢMissing At

    Randomʣ ‣ ܽଛม਺ࣗମʹґଘͯܽ͠ଛˠNMAR ʢNot Missing At Randomʣ ܽଛλΠϓ
  23. ‣ ׬શʹϥϯμϜʹܽଛˠMCAR ʢMissing Completely At Randomʣ ‣ ؍ଌม਺ʹґଘͯܽ͠ଛˠMAR ʢMissing At

    Randomʣ ‣ ܽଛม਺ࣗମʹґଘͯܽ͠ଛˠNMAR ʢNot Missing At Randomʣ ରॲࠔ೉ ܽଛλΠϓ
  24. ‣ ׬શʹϥϯμϜʹܽଛˠMCAR ʢMissing Completely At Randomʣ ‣ ؍ଌม਺ʹґଘͯܽ͠ଛˠMAR ʢMissing At

    Randomʣ ‣ ܽଛม਺ࣗମʹґଘͯܽ͠ଛˠNMAR ʢNot Missing At Randomʣ ܽଛλΠϓ
  25. ‣ ׬શʹϥϯμϜʹܽଛˠMCAR ʢMissing Completely At Randomʣ ‣ ؍ଌม਺ʹґଘͯܽ͠ଛˠMAR ʢMissing At

    Randomʣ ‣ ܽଛม਺ࣗମʹґଘͯܽ͠ଛˠNMAR ʢNot Missing At Randomʣ ܽଛλΠϓ
  26. 1. ফ͢ 2. ຒΊΔ 1. ফ͢ 2. ຒΊΔ ୯Ұ୅ೖ๏

  27. ͲΜͳ஋ͰຒΊΔͷ͔ʁ ୯Ұ୅ೖ๏

  28. 1. ฏۉ஋ʢฏۉ஋୅ೖ๏ʣ 2. ༧ଌ஋ʢ֬ఆతճؼ୅ೖ๏ʣ 3. ༧ଌ஋ + ࢒ࠩʢ֬཰తճؼ୅ೖ๏ʣ + ୯Ұ୅ೖ๏

  29. 1. ฏۉ஋ʢฏۉ஋୅ೖ๏ʣ 2. ༧ଌ஋ʢ֬ఆతճؼ୅ೖ๏ʣ 3. ༧ଌ஋ + ࢒ࠩʢ֬཰తճؼ୅ೖ๏ʣ + ୯Ұ୅ೖ๏

  30. ฏۉ஋Λܭࢉ͠୅ೖ ୯Ұ୅ೖ๏

  31. ฏۉ஋Λܭࢉ͠୅ೖ ฏۉ஋୅ೖ๏ ฏۉ஋ ≒ உੑͷମॏͷฏۉ ୯Ұ୅ೖ๏

  32. 1. ฏۉ஋ʢฏۉ஋୅ೖ๏ʣ 2. ༧ଌ஋ʢ֬ఆతճؼ୅ೖ๏ʣ 3. ༧ଌ஋ + ࢒ࠩʢ֬཰తճؼ୅ೖ๏ʣ + ୯Ұ୅ೖ๏

  33. ܽଌ஋Λ༧ଌ͢ΔϞσϧΛ࡞Δ *ิॿม਺ ܽ ଛ ͷ ͋ Δ ม ਺ *ิॿม਺:ܽଛ஋ͷ༧ଌʹ࢖ΘΕΔม਺ɻඞͣ͠΋໨తม਺ͷ༧ଌʹ࢖ΘΕΔΘ͚Ͱ͸ͳ͍ɻ

    ୯Ұ୅ೖ๏
  34. ܽଌ஋Λ༧ଌ͢ΔϞσϧΛ࡞Δ ิॿม਺ ܽ ଛ ͷ ͋ Δ ม ਺ ୯Ұ୅ೖ๏

  35. ܽଌ஋Λ༧ଌ͢ΔϞσϧΛ࡞Δ ܽ ଛ ͷ ͋ Δ ม ਺ ʢ֬ఆతʣ ճؼ୅ೖ๏

    ิॿม਺ ୯Ұ୅ೖ๏
  36. ܽ ଛ ͷ ͋ Δ ม ਺ ิॿม਺ ୅ೖ஋͕શͯ௚ઢ্ʹ ୯Ұ୅ೖ๏

  37. ܽ ଛ ͷ ͋ Δ ม ਺ ༧ଌ஋ͷޡࠩΛաখධՁ ิॿม਺ ୅ೖ஋͕શͯ௚ઢ্ʹ

    ୯Ұ୅ೖ๏
  38. 1. ฏۉ஋ʢฏۉ஋୅ೖ๏ʣ 2. ༧ଌ஋ʢ֬ఆతճؼ୅ೖ๏ʣ 3. ༧ଌ஋ + ࢒ࠩʢ֬཰తճؼ୅ೖ๏ʣ 1. ฏۉ஋ʢฏۉ஋୅ೖ๏ʣ

    2. ༧ଌ஋ʢ֬ఆతճؼ୅ೖ๏ʣ 3. ༧ଌ஋ + ࢒ࠩʢ֬཰తճؼ୅ೖ๏ʣ + ୯Ұ୅ೖ๏
  39. ୅ೖϞσϧʹޡ߲ࠩΛ௥Ճ ܽ ଛ ͷ ͋ Δ ม ਺ ิॿม਺ ޡ߲ࠩ

    ୯Ұ୅ೖ๏
  40. ܽ ଛ ͷ ͋ Δ ม ਺ ิॿม਺ ୅ೖϞσϧʹޡ߲ࠩΛ௥Ճ ͹Β͖ͭΛ൓ө

    ୯Ұ୅ೖ๏
  41. ܽ ଛ ͷ ͋ Δ ม ਺ ิॿม਺ ୅ೖϞσϧʹޡ߲ࠩΛ௥Ճ ֬཰తճؼ୅ೖ๏

    ୯Ұ୅ೖ๏
  42. ୅ೖϞσϧͷޡ߲ࠩ͸൓өग़དྷͨ ܽ ଛ ͷ ͋ Δ ม ਺ ิॿม਺ ୯Ұ୅ೖ๏

  43. ୅ೖϞσϧͦͷ΋ͷͷෆ࣮֬ੑ͸ʁ ܽ ଛ ͷ ͋ Δ ม ਺ ิॿม਺ ʁ

    ʁ ʁ ୯Ұ୅ೖ๏
  44. 1ͭͷ஋Λ୅ೖʢ୯Ұ୅ೖ๏ʣ ෳ਺ͷ஋Λ୅ೖ ʢଟॏ୅ೖ๏ʣ ଟॏ୅ೖ๏

  45. ୅ೖ ෼ੳ ౷߹ ෳ਺ͷ୅ೖ஋Λਪఆ͠ෳ਺ͷ୅ೖࡁΈσʔλΛੜ੒ ୅ೖࡁΈσʔλΛ࢖༻ͯ͠฼਺ਪఆ ਪఆ஋Λ౷߹͠ɺ࠷ऴ݁Ռͱ͢Δ ଟॏ୅ೖ๏

  46. ܽଛσʔλ ୅ೖࡁΈσʔλ1 ୅ೖࡁΈσʔλ2 ୅ೖࡁΈσʔλ3 ෼ੳ݁Ռ1 ෼ੳ݁Ռ2 ෼ੳ݁Ռ3 ࠷ऴ݁Ռ ୅ೖ ෼ੳ

    ౷߹ ଟॏ୅ೖ๏
  47. ܽଛσʔλ ୅ೖࡁΈσʔλ1 ୅ೖࡁΈσʔλ2 ୅ೖࡁΈσʔλ3 ෼ੳ݁Ռ1 ෼ੳ݁Ռ2 ෼ੳ݁Ռ3 ࠷ऴ݁Ռ ୅ೖ ෼ੳ

    ౷߹ ଟॏ୅ೖ๏
  48. ‣ DA๏ : Data Augmentationʢσʔλ֦େ๏ʣ ‣ FCS๏ : Fully Conditional

    Specificationʢ׬ શ৚݅෇͖ࢦఆʣ ‣ EMB๏ : Expectation-Maximization with Bootstrapping ଟॏ୅ೖ๏
  49. ‣ ܽଛআڈɿϖΞϫΠζ๏ɺม਺আڈ ‣ ୯Ұ୅ೖ๏ɿൺ཰୅ೖ๏ɺϗοτσοΫ๏ɺίʔ ϧυσοΫ๏ɺLOCFɺNOCB ‣ ଟॏ୅ೖ๏ɿ୅ೖ਍அɺײ౓෼ੳ ‣ ׬શ৘ใ࠷໬๏ ‣

    LightGBMͳͲͷܽଛ஋ࣗಈॲཧ ࠓճ঺հ͠ͳ͔ͬͨ΋ͷ
  50. ‣ ߴڮকٓ,౉ลඒஐࢠ, ܽଌσʔλॲཧ: RʹΑΔ୯Ұ୅ೖ๏ͱ ଟॏ୅ೖ๏ (౷ܭֶOne Point), ڞཱग़൛, 2017/12/9, 192ϖʔ

    δ ‣ ܽଛ஋͕͋Δσʔλͷ෼ੳ, Sunny side up!, http:// norimune.net/1811 ‣ RͰ࣮ફʂܽଛσʔλ෼ੳೖ໳ʲ1ʳ, NHN TECHORUS Tech Blog, https://techblog.nhn-techorus.com/archives/6573 ‣ R ܽଛ஋ͷରԠʢmissing value treatmentʣ, ౷ܭֶͱӸֶͱ ࣌ʑɺॿڭੜ׆, http://jojoshin.hatenablog.com/entry/ 2017/02/03/220118 ࢀߟจݙ
  51. ENJOY! A