About Missing Values - Speaker Deck

Slide 1

Slide 1 text

ܽଛ஋ॲཧʹ͍ͭͯ dࣦΘΕͨ஋ΛٻΊͯd

Slide 2

Slide 2 text

໨࣍ 1. ܽଛ஋ͱ͸ 2. ܽଛআڈ 3. ܽଛλΠϓ 4. ୯Ұ୅ೖ๏ 5. ଟॏ୅ೖ๏ 6. ࠓճ঺հ͠ͳ͔ͬͨ΋ͷ 7. ࢀߟจݙ

Slide 3

Slide 3 text

ܽଛ஋ͱ͸ Missing Valueʢܽଛ஋ɺܽଌ஋ʣ In statistics, missing data, or missing values, occur when no data value is stored for the variable in an observation. Missing data are a common occurrence and can have a signiﬁcant effect on the conclusions that can be drawn from the data. ʢhttps://en.wikipedia.org/wiki/Missing_dataʣ

Slide 4

Slide 4 text

ܽଛ͕͋Δͱ Կ͕ࠔΔͷ͔ʁ ܽଛ஋ͱ͸

Slide 5

Slide 5 text

ܭࢉग़དྷͳ͍ Q,10, 20, ܽଛ஋ͷฏۉ͸ʁ ܽଛ஋ͱ͸ > mean(c(10, 20, NA)) [1] NA

Slide 6

Slide 6 text

Ͳ͏͢Δͷ͔ʁ ܽଛ஋ͱ͸

Slide 7

Slide 7 text

1. ফ͢ 2. ຒΊΔ ܽଛ஋ͱ͸

Slide 8

Slide 8 text

1. ফ͢ 2. ຒΊΔ 1. ফ͢ ܽଛ஋ͱ͸

Slide 9

Slide 9 text

ܽଛͷ͋ΔϨίʔυΛফ͢ ମॏ ੑผ 50 ঁ 70 உ 55 NA NA உ ܽଛ஋আڈ

Slide 10

Slide 10 text

ମॏ ੑผ 50 ঁ 70 உ 55 NA NA உ ମॏ ੑผ 50 ঁ 70 உ ܽଛͷ͋ΔϨίʔυΛফ͢ ܽଛ஋আڈ

Slide 11

Slide 11 text

ମॏ ੑผ 50 ঁ 70 உ 55 NA NA உ ମॏ ੑผ 50 ঁ 70 உ ϦετϫΠζ๏ ܽଛͷ͋ΔϨίʔυΛফ͢ ܽଛ஋আڈ

Slide 12

Slide 12 text

ফ͍͍ͯ͠ͷʁ 1. ফͯ͠΋ྑ͍৔߹ 2. ফͯ͠͸ବ໨ͳ৔߹ ܽଛ஋আڈ

Slide 13

Slide 13 text

׬શʹϥϯμϜʹܽଛ ܽଛύλʔϯ͸ੑผ ΍ମॏͱແؔ܎ ܽଛ஋আڈ

Slide 14

Slide 14 text

׬શʹϥϯμϜʹܽଛ ܽଛύλʔϯ͸ੑผ ΍ମॏͱແؔ܎ ফͯ͠΋໰୊͸ͳ͍ ͨͩ͠ɺޮ཰͸Լ͕Δ ܽଛ஋আڈ

Slide 15

Slide 15 text

؍ଌม਺ʹґଘͯܽ͠ଛ ঁੑͷํ͕ଟܽ͘ଛ ܽଛ஋আڈ

Slide 16

Slide 16 text

؍ଌม਺ʹґଘͯܽ͠ଛ ফ͢ͱঁੑͷσʔλͷΈݮগ ภΔ ঁੑͷํ͕ଟܽ͘ଛ ܽଛ஋আڈ

Slide 17

Slide 17 text

ܽଛม਺ࣗମʹґଘͯܽ͠ଛ ମॏͷॏ͍ํ͕ଟ͘ ܽଛ ܽଛ஋আڈ

Slide 18

Slide 18 text

ܽଛม਺ࣗମʹґଘͯܽ͠ଛ ফ͢ͱମॏͷߴ͍σʔλ͕ݮগ ภΔ ମॏͷॏ͍ํ͕ଟ͘ ܽଛ ܽଛ஋আڈ

Slide 19

Slide 19 text

‣ ׬શʹϥϯμϜʹܽଛˠMCAR ʢMissing Completely At Randomʣ ‣ ؍ଌม਺ʹґଘͯܽ͠ଛˠMAR ʢMissing At Randomʣ ‣ ܽଛม਺ࣗମʹґଘͯܽ͠ଛˠNMAR ʢNot Missing At Randomʣ ܽଛλΠϓ

Slide 20

Slide 20 text

Slide 21

Slide 21 text

‣ ׬શʹϥϯμϜʹܽଛˠMCAR ʢMissing Completely At Randomʣ ‣ ؍ଌม਺ʹґଘͯܽ͠ଛˠMAR ʢMissing At Randomʣ ‣ ܽଛม਺ࣗମʹґଘͯܽ͠ଛˠNMAR ʢNot Missing At Randomʣ ফͯ͠΋ྑ͍ ܽଛλΠϓ

Slide 22

Slide 22 text

Slide 23

Slide 23 text

‣ ׬શʹϥϯμϜʹܽଛˠMCAR ʢMissing Completely At Randomʣ ‣ ؍ଌม਺ʹґଘͯܽ͠ଛˠMAR ʢMissing At Randomʣ ‣ ܽଛม਺ࣗମʹґଘͯܽ͠ଛˠNMAR ʢNot Missing At Randomʣ ରॲࠔ೉ ܽଛλΠϓ

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Slide 26

Slide 26 text

1. ফ͢ 2. ຒΊΔ 1. ফ͢ 2. ຒΊΔ ୯Ұ୅ೖ๏

Slide 27

Slide 27 text

ͲΜͳ஋ͰຒΊΔͷ͔ʁ ୯Ұ୅ೖ๏

Slide 28

Slide 28 text

1. ฏۉ஋ʢฏۉ஋୅ೖ๏ʣ 2. ༧ଌ஋ʢ֬ఆతճؼ୅ೖ๏ʣ 3. ༧ଌ஋ + ࢒ࠩʢ֬཰తճؼ୅ೖ๏ʣ + ୯Ұ୅ೖ๏

Slide 29

Slide 29 text

1. ฏۉ஋ʢฏۉ஋୅ೖ๏ʣ 2. ༧ଌ஋ʢ֬ఆతճؼ୅ೖ๏ʣ 3. ༧ଌ஋ + ࢒ࠩʢ֬཰తճؼ୅ೖ๏ʣ + ୯Ұ୅ೖ๏

Slide 30

Slide 30 text

ฏۉ஋Λܭࢉ͠୅ೖ ୯Ұ୅ೖ๏

Slide 31

Slide 31 text

ฏۉ஋Λܭࢉ͠୅ೖ ฏۉ஋୅ೖ๏ ฏۉ஋ ≒ உੑͷମॏͷฏۉ ୯Ұ୅ೖ๏

Slide 32

Slide 32 text

1. ฏۉ஋ʢฏۉ஋୅ೖ๏ʣ 2. ༧ଌ஋ʢ֬ఆతճؼ୅ೖ๏ʣ 3. ༧ଌ஋ + ࢒ࠩʢ֬཰తճؼ୅ೖ๏ʣ + ୯Ұ୅ೖ๏

Slide 33

Slide 33 text

ܽଌ஋Λ༧ଌ͢ΔϞσϧΛ࡞Δ *ิॿม਺ ܽ ଛ ͷ ͋ Δ ม ਺ *ิॿม਺:ܽଛ஋ͷ༧ଌʹ࢖ΘΕΔม਺ɻඞͣ͠΋໨తม਺ͷ༧ଌʹ࢖ΘΕΔΘ͚Ͱ͸ͳ͍ɻ ୯Ұ୅ೖ๏

Slide 34

Slide 34 text

ܽଌ஋Λ༧ଌ͢ΔϞσϧΛ࡞Δ ิॿม਺ ܽ ଛ ͷ ͋ Δ ม ਺ ୯Ұ୅ೖ๏

Slide 35

Slide 35 text

ܽଌ஋Λ༧ଌ͢ΔϞσϧΛ࡞Δ ܽ ଛ ͷ ͋ Δ ม ਺ ʢ֬ఆతʣ ճؼ୅ೖ๏ ิॿม਺ ୯Ұ୅ೖ๏

Slide 36

Slide 36 text

ܽ ଛ ͷ ͋ Δ ม ਺ ิॿม਺ ୅ೖ஋͕શͯ௚ઢ্ʹ ୯Ұ୅ೖ๏

Slide 37

Slide 37 text

ܽ ଛ ͷ ͋ Δ ม ਺ ༧ଌ஋ͷޡࠩΛաখධՁ ิॿม਺ ୅ೖ஋͕શͯ௚ઢ্ʹ ୯Ұ୅ೖ๏

Slide 38

Slide 38 text

1. ฏۉ஋ʢฏۉ஋୅ೖ๏ʣ 2. ༧ଌ஋ʢ֬ఆతճؼ୅ೖ๏ʣ 3. ༧ଌ஋ + ࢒ࠩʢ֬཰తճؼ୅ೖ๏ʣ 1. ฏۉ஋ʢฏۉ஋୅ೖ๏ʣ 2. ༧ଌ஋ʢ֬ఆతճؼ୅ೖ๏ʣ 3. ༧ଌ஋ + ࢒ࠩʢ֬཰తճؼ୅ೖ๏ʣ + ୯Ұ୅ೖ๏

Slide 39

Slide 39 text

୅ೖϞσϧʹޡ߲ࠩΛ௥Ճ ܽ ଛ ͷ ͋ Δ ม ਺ ิॿม਺ ޡ߲ࠩ ୯Ұ୅ೖ๏

Slide 40

Slide 40 text

ܽ ଛ ͷ ͋ Δ ม ਺ ิॿม਺ ୅ೖϞσϧʹޡ߲ࠩΛ௥Ճ ͹Β͖ͭΛ൓ө ୯Ұ୅ೖ๏

Slide 41

Slide 41 text

ܽ ଛ ͷ ͋ Δ ม ਺ ิॿม਺ ୅ೖϞσϧʹޡ߲ࠩΛ௥Ճ ֬཰తճؼ୅ೖ๏ ୯Ұ୅ೖ๏

Slide 42

Slide 42 text

୅ೖϞσϧͷޡ߲ࠩ͸൓өग़དྷͨ ܽ ଛ ͷ ͋ Δ ม ਺ ิॿม਺ ୯Ұ୅ೖ๏

Slide 43

Slide 43 text

୅ೖϞσϧͦͷ΋ͷͷෆ࣮֬ੑ͸ʁ ܽ ଛ ͷ ͋ Δ ม ਺ ิॿม਺ ʁ ʁ ʁ ୯Ұ୅ೖ๏

Slide 44

Slide 44 text

1ͭͷ஋Λ୅ೖʢ୯Ұ୅ೖ๏ʣ ෳ਺ͷ஋Λ୅ೖ ʢଟॏ୅ೖ๏ʣ ଟॏ୅ೖ๏

Slide 45

Slide 45 text

୅ೖ ෼ੳ ౷߹ ෳ਺ͷ୅ೖ஋Λਪఆ͠ෳ਺ͷ୅ೖࡁΈσʔλΛੜ੒ ୅ೖࡁΈσʔλΛ࢖༻ͯ͠฼਺ਪఆ ਪఆ஋Λ౷߹͠ɺ࠷ऴ݁Ռͱ͢Δ ଟॏ୅ೖ๏

Slide 46

Slide 46 text

ܽଛσʔλ ୅ೖࡁΈσʔλ1 ୅ೖࡁΈσʔλ2 ୅ೖࡁΈσʔλ3 ෼ੳ݁Ռ1 ෼ੳ݁Ռ2 ෼ੳ݁Ռ3 ࠷ऴ݁Ռ ୅ೖ ෼ੳ ౷߹ ଟॏ୅ೖ๏

Slide 47

Slide 47 text

ܽଛσʔλ ୅ೖࡁΈσʔλ1 ୅ೖࡁΈσʔλ2 ୅ೖࡁΈσʔλ3 ෼ੳ݁Ռ1 ෼ੳ݁Ռ2 ෼ੳ݁Ռ3 ࠷ऴ݁Ռ ୅ೖ ෼ੳ ౷߹ ଟॏ୅ೖ๏

Slide 48

Slide 48 text

‣ DA๏ : Data Augmentationʢσʔλ֦େ๏ʣ ‣ FCS๏ : Fully Conditional Speciﬁcationʢ׬ શ৚݅෇͖ࢦఆʣ ‣ EMB๏ : Expectation-Maximization with Bootstrapping ଟॏ୅ೖ๏

Slide 49

Slide 49 text

‣ ܽଛআڈɿϖΞϫΠζ๏ɺม਺আڈ ‣ ୯Ұ୅ೖ๏ɿൺ཰୅ೖ๏ɺϗοτσοΫ๏ɺίʔ ϧυσοΫ๏ɺLOCFɺNOCB ‣ ଟॏ୅ೖ๏ɿ୅ೖ਍அɺײ౓෼ੳ ‣ ׬શ৘ใ࠷໬๏ ‣ LightGBMͳͲͷܽଛ஋ࣗಈॲཧ ࠓճ঺հ͠ͳ͔ͬͨ΋ͷ

Slide 50

Slide 50 text

‣ ߴڮকٓ,౉ลඒஐࢠ, ܽଌσʔλॲཧ: RʹΑΔ୯Ұ୅ೖ๏ͱ ଟॏ୅ೖ๏ (౷ܭֶOne Point), ڞཱग़൛, 2017/12/9, 192ϖʔ δ ‣ ܽଛ஋͕͋Δσʔλͷ෼ੳ, Sunny side up!, http:// norimune.net/1811 ‣ RͰ࣮ફʂܽଛσʔλ෼ੳೖ໳ʲ1ʳ, NHN TECHORUS Tech Blog, https://techblog.nhn-techorus.com/archives/6573 ‣ R ܽଛ஋ͷରԠʢmissing value treatmentʣ, ౷ܭֶͱӸֶͱ ࣌ʑɺॿڭੜ׆, http://jojoshin.hatenablog.com/entry/ 2017/02/03/220118 ࢀߟจݙ

Slide 51

Slide 51 text

ENJOY! A