An Introduction to Statistical Learning 4章

An Introduction to Statistical Learningの輪読を行ったときに作成したスライドです。

Transcript

1. 4 Classification 0Ő
2. â¾ 4.1 An Overview of Classification 4.2 Why Not Linear Regression? 4.3 Logistic Regression 4.3.1 The Logistic Model 4.3.2 Estimating the Regression Coefficients 4.3.3 Making Predictions 4.3.4 Multiple Logistic Regression 4.3.5 Logistic Regression for >2 Response Classes 4.4 Linear Discriminant Analysis 4.4.1 Using Bayes’ Theorem for Classification 4.4.2 Linear Discriminant Analysis for p=1 4.4.3 Linear Discriminant Analysis for p>1 4.4.4 Quadratic Discriminant Analysis 4.5 A Comparison of Classification Methods 4.6 Lab: Logistic Regression, LDA, QDA, and KNN 4.7 Exercises 2
3. 4 Classification ĂRsƻƪƿŹ3ðŲġ§Ťūƌ ũŸųŞŔâá\¢ŹĺáŲŗŮūƌŷ ŤŜŤŔ×gOōŲŹâá\¢ŝĥáŸYHƇ ŗƏƌ ŢŸðŲŹŔĥáŵâá\¢Ÿ ÍŸ¦ÈŶů Řűġ§ŦƏƌ ŢŸŢųƓ0Ő(Classification)ųLſƌ 3
4. 4.1 An Overview of Classification 0ŐŸºĔ
5. 4.1 An Overview of Classification ƝǀƢƨƫƛDŽƬŸDefaultƪDŽƦ(100000)Ŭƌ IncomeŹuVvBŔBalanceŹ®ŸĢÅŎŬƌ ŇŹŭƈƔų–Ůū(default=No)ŲŔƚǀǂƢ Ź–ŮűŵŘ(default=Yes)Ŭƌ 5
6. 4.1 An Overview of Classification Balance(!" )ųIncome(!# )ŜƍDefault($)Ɠ ÍŦ Ə•ÈƓ4ðŲġ§ŦƏƌ $Źĺá\¢ŲŹŵŘŜƍŔ3ðŲġ§ŤūĂ RsŹ ŚŵŘƌ 6
7. 4.2 Why Not Linear Regression? ĂRsťƈŬƆŵŸLJ
8. 4.2 Why Not Linear Regression? ŵƔŲĂRsŹĥá\¢ŶŹ ŚŵŘŸLJ ˆ MhŸŠĈŸàÕŸƪDŽƦŶĂRsƓÐØ ƉƎ ŮűƅƏƌ âá\¢$ƓŸ»ŶfĆŦƏƌ 8
9. 4.2 Why Not Linear Regression? ŢŸfĆŹŔstrokeųepileptic seizureŸ ļŶdrug overdoseƓśŘűƏƌ ũƐŹŔstrokeųdrug overdoseŸqųdrug overdoseų epileptic seizureŸqŝòŤŘŢųƓŽKŤűŘƏƌŷ ŠŴgŃŹũŸƌřŵ gŹŵŘƌ 9
10. 4.2 Why Not Linear Regression? ġ§\¢Ÿ&ŝŔĩxŔ ìxŔĹxųŋwŠƓ ŤűŔĩxŜƍ ìxŸqŝ ìxŜƍĹxŸqų òŤŘųťūYHŹŔ1,2,3ųŘřfĆŹŘŘŜƇŤ ƐŵŘƌ ůŸÕƓųƏĥáŵâá\¢ŸĂRs Ÿ¦ÈŹ
    ČáŶŹŵŘƌ ůŸÕŸYHŹŘئÈŝŗƏƌ 10
11. 4.2 Why Not Linear Regression? ŢŸƌřŶfĆŤŵśŦƌ ŢŸYHŹĂRsƓ řŢųŝŲŞƏƌ ŬŠŴŔ$ŝ[0,1]ŸõS]Ŷ×ƐūƎŤűŔèÖųŤ űęĸŲŞŵşŵƎƌşŵŘƌ ċKÊŘŢųŶŢŸYHŹŔ4.4ôŲġ§ŦƏĂ3 40²(LDA)ųIťû³ŝ„ƍƐƏƌ 11
12. 4.3 Logistic Regression ǁƢƣƩƕƨƝRs
13. 4.3 Logistic Regression ƝǀƢƨƫƛDŽƬŸdefaultƪDŽƦŶ“ŮűćŚƏƌ âá\¢Źdefault=Yes,NoŸůŶ0ŠƍƐƏƌ ǁƢƣƩƕƨƝRsŹŢŸâá\¢$Ɠãœƻƪƿ :ŦƏƔťƈŵşű$ŝÔfŸƛƩƟƾŶoŦƏèÖ Ɠƻƪƿ:ŦƏƌ 13
14. 4.3 Logistic Regression pŹĂRsƓ Ůūƌ èÖŝƷƖƭƣŶŵŮűƏƌDž FŹǁƢƣƩƕƨƝRsƓ Ůūƌ ,ŌWŲ[0,1]ŶBƄŮűƏƌDž 14
15. 4.3 Logistic Regression balanceŶƌƏdefaultŸèÖŹŸ»ŶŜŠƏƌ ŬŠŴåÜŤű ų¬şƌ blanceŸŽŸ&ŶƌŮűdefaultŸèÖƓ ÍŲŞ Əƌ ¹ŝdefaultŸƓ‘ĹŶ ÍŦƏYH ŸƌřŘŶŘľ&ƓĝfŦƏŢųƇŲŞƏƌ 15
16. 4.3.1 The Logistic Model ǁƢƣƩƕƨƝƻƪƿ
17. 4.3.1 The Logistic Model ŴřƉŮű!ųp(!)ŸĽ$Ɠƻƪƿ:ŤūƍŘŘŸLJ ©ıŸĂRsŶƌƏ ÍŲŹƧƺŬƌ ŴƔŵbalanceŸ&ŲƇdefaultŸèÖŹ[0,1]Ÿ ŶB ƄŮű¿ŤŘƌ ŢŸOōŹƁůŶdefaultƪDŽƦŬŠŸOōťƈŵŘƌ âá\¢ŝ2&ŸYHŶRs0²ƓđřųŔ?ØáŶ Źp ! > 1ƇŤşŹp ! < 0ųŵƏYHŝ…ŧŗƏŜ ƍŷ 17
18. 4.3.1 The Logistic Model ŢŸOōƓęÆŦƏŶŹ,űŸ!ŶśŘűp ! Ÿ&ŝ [0,1]ŶBƄƏƌřŵĽ¢Ɠ Ůűp ! Ɠƻƪƿ:ŤŵŠ ƐźŘŠŵŘŷ ũŸƌřŵĽ¢ŹƇŭƑƔŘŮŻŘŗƏƌ ǁƢƣƩƕƨƝRsŲŹǁƢƣƩƕƨƝĽ¢Ɠ řƌ 18
19. 4.3.1 The Logistic Model ŢŸƻƪƿƓ~űŹƆƏūƆŶ­mÈƓ řƌ ­mÈŹ¾ŸƥƝơƼǂŲġ§ŦƏƌ ţŮŞŸT(F)ƇǁƢƣƩƕƨƝĽ¢Ɠ~űŹƆūƌ defaultŸèÖŹĿƎŵş0ŶĭŰŘűŘƏŠŴ0Ŷ ŹŵŮűŵŘƌ balanceŝŒŘYHƇI»Ŭƌ ǁƢƣƩƕƨƝĽ¢ŹŴƔŵųŞƇSb Ɠžşƌ 19
20. 4.3.1 The Logistic Model lŤţŮŞŸyƓy\ŦƏƌ 20
21. 4.3.1 The Logistic Model pĬƓƚƨƤųĚřƌ ƚƨƤŹñőŬŠťƈŵşűŘƑŘƑŵųŢƑŶ ƒƐ űŞūƌ ƚƨƤŹ0Ŝƍ∞Ÿ&ƓųƏƌ 21
22. 4.3.1 The Logistic Model Ĭj¢ųŮūƌ pĬŹǁƢƨƫųĚřƌ ŢřŵƏų!ŶjŤűĂŬŷ 22
23. 4.3.1 The Logistic Model 3ðŸĂRsƻƪƿŲŹ-" Ź!Ÿ<[8ŶjŦƏ $Ÿ[8ĺƓĒŤűŘūŷ ǁƢƣƩƕƨƝRsƻƪƿŸYHŹũřŲŹŵŘƌ !Ÿ<[8ŶjŦƏp ! Ÿ[8ĺŹ!Ÿ&Ŷ"cŦ Əƌ ŬŠŴ!ŝ[8ŦƐźp ! Ƈ[8Ŕ !ŝÌlŦƐźp ! ƇÌlųŘřĽ $Ź’Ǝïůƌ 23
24. 4.3.2 Estimating the Regression Coefficient Rs$¢Ÿf
25. 4.3.2 Estimating the Regression Coefficient -. ų-" ŹĜăƪDŽƦŜƍfŤŵŠƐźŘŠŵŘƌ 3ðŲŹ$¢ŸfŶ­k ÈƓ Ůūƌ ŢŸYHƇňĂŸ­k ÈƓ řŢųƇŲŞƏŠŴ ­mÈŸ¦ŝŘŘƌ 25
26. 4.3.2 Estimating the Regression Coefficient ŢŸĽ¢ŹmxĽ¢ųŘřƌ mxĽ¢Ÿ&ŝ­_ųŵƏƌřŶ-. ų-" ƓĕůŠƏƌ ­mÈŸ¢dáŵęġŹŢŸ°ŲŹ—ƒŵŘƌ ǁƢƣƩƕƨƝRsŸƻƪƿŸ~űŹƆŹRƓ Śźö <ŶŲŞƏƌ 26
27. 4.3.2 Estimating the Regression Coefficient RŲǁƢƣƩƕƨƝRsƓŤūû³ƓéŦƌ ĕ¦Ź3ðŸĂRsųIťŬƌ ZýěĺŹtýěĺųIť7Ŭƌ !Śź-" ŶĽŦƏZŸ&Ź- / "/SE(- / ")Ŭƌ ŢŸ&Ÿ_ŞţŹ(þj&)ŔsÐġ3.: -" = 0Ɠ¶> ŲŞƏŜŴřŜŶĽ$ŦƏƌ 27
28. 4.3.2 Estimating the Regression Coefficient 3.: -" = 0ųŘřsÐġŹ ŲŗƏŢųƓéŦƌ ŢƐŹdefaultŸèÖŹbalanceŸ&ŶƌƍŵŘŢųƓĒ ŤűŘƏƌ ŬŠŴp&ŝųűƇkţŘŜƍ3. Ź¶>ŲŞƏŷ 28
29. 4.3.3 Making Predictions ÍŦƏ
30. 4.3.3 Making Predictions $¢ŝfŲŞƐźŽŸbalanceŶjŤűdefaultŸ èÖƓěóŦƏŸŹö<Ŭƌ !Śźbalanceŝ1000$ŸųŞŸdefaultŸèÖŹ ų1%Ŭƌ ŬŠŴ2000$ŲěóŦƏų58.6%ŶĦŷŝƏƌ 30
31. 4.3.3 Making Predictions ġ§\¢ŝƧƸDŽ\¢ŲƇ_`Ŭƌ defaultƪDŽƦŶŹdÙŜŴřŜŸŒXƇŗƏƌ dÙŬų1ŔňdÙŬų0ųŤűǁƢƣƩƕƨƝRsŤūû ³ƓĒŶéŦƌ studentŸ$¢ŹÀŲp&Ź¯ŽŬƌ 31
32. 4.3.4 Multiple Logistic Regression ^ĹǁƢƣƩƕƨƝRs
33. 4.3.4 Multiple Logistic Regression ¾ŹŔē¢Ÿġ§\¢Ŝƍ0,1Ÿâá\¢Ɠ ÍŦƏ OōŶůŘűćŚƏƌ ţŮŞŸyƓš|ŦƏƌ ţŮŞųİŸy\ƓŦƏƌ -., -" ,,,, -7 Ÿf¦ÈƇţŮŞųIťŲ­mÈƓ řƌ 33
34. 4.3.4 Multiple Logistic Regression balance,income,studentƓġ§\¢ŔdefaultƓâá\¢ ųŤűǁƢƣƩƕƨƝRsƓ ÚŤūû³ƓéŦƌ balanceųstudentŸp&ŝkţŘŸŲdefaultųĽ$Ťű ŘƏƌ studentŸ$¢ŝƷƖƭƣŵŸŹdÙŸƂřŝňdÙ ŶÃƁűdefaultŤŶşŘŢųƓĒŤűŘƏƌ ţŮŞŸĒųİŸû³ƓéŤűŘƏŷ ŵƔųŘřŢųŲŤƋř 34
35. 4.3.4 Multiple Logistic Regression ŢŸTƓ Ůűġ§ŤűŘşƌ ƚǀǂƢŝdÙŲŇŝňdÙŬƌ 35
36. 4.3.4 Multiple Logistic Regression pŸTŸgĂŹuVdefaultÖƓbalanceŸĽ¢ųŤű ĒŤűŘƏƌ ^ĹǁƢƣƩƕƨƝRsŸû³ŹŔŗƏbalanceųincome ŶjŤűdÙŹňdÙƌƎdefaultŤŶşŘŢųƓéŤű Řūƌŷ pŸTŸgĂŹèŜŶũřŵŮűŘƏƌŷ 36
37. 4.3.4 Multiple Logistic Regression pŸTŸÄuçĂŹŔdÙųňdÙŸuVdefaultÖƓ éŤűŘƏƌ û³ŝİŶŵŮūŷ 37
38. 4.3.4 Multiple Logistic Regression FŸTŲŢŸ
    Ċŝġ§ŲŞƏƌ sutudentųbalanceŹäĽŤűŘƏŷ dÙŹbalanceŝŒŘ(JŶŗƎŢƐŹdefaultŶůŵ ŝƏƌ 38
39. 4.3.4 Multiple Logistic Regression IťbalanceŸYHŔdÙŹňdÙŶÃƁűdefaultÖ ŹŘƌ ŤŜŤŔdÙŹ,áŶbalanceŝŒŘ(JŶŗƎŔ defaultÖŹŒŘƌ 39
40. 4.3.4 Multiple Logistic Regression ŢŸIJŘŹƝǀƢƨƫƛDŽƬêŶųŮűŹ_ŞŘƌ ƪDŽƦŝŵŘdÙŹŔňdÙƌƎƇ=ŁŬƌ ŬŠŴũŸdÙŹŔIťbalanceƓ›ůňdÙƌƎŜŹ =Ł‰ŝŘƌ ŢŸƌřŵ×ģƓü(confounding)ųĚřƌ 40
41. 4.3.5 Logistic Regression for >2 Response Class 3ƝƽƣŸǁƢƣƩƕƨƝRs
42. 4.3.5 Logistic Regression for >2 Response Class 6ŸƥƝơƼǂŲġ§Ťū2ƝƽƣŸǁƢƣƩƕƨƝRs ƻƪƿŹ3ƝƽƣŶƇš|ŲŞƏƌ ŲƇgŃŶŹŗƄƎ ƒŵŘƌ ŵƔŲŜųŘřųŔ¾ŸƥƝơƼǂŲġ§ŦƏ340² ŝē¢ƝƽƣŸ0ŐŲ
    ČáŬŜƍŬƌ ŬŜƍŢŸ°ŲŹġ§ŤŵŘŠŴũřŘřŸŝŗƏŮű ŢųŬŠėŚűśŘűŷ ŭŵƅŶRŲŲŞƏƌ 42
43. 4.4 Linear Discriminant Analysis Ă340²
44. 4.4 Linear Discriminant Analysis ǁƢƣƩƕƨƝRsųŹ4Ÿ•ÈƓćŚƏƌ ũŸ•ÈŸû³ŹǁƢƣƩƕƨƝRsųňtŶƌşű ŘƏƌ ťƈŗŵƔŲǁƢƣƩƕƨƝRsŝŗƏŸŶ4Ÿ•ÈƓ ćŚƏŸLJ 44
45. 4.4 Linear Discriminant Analysis ũƐŹ ǃƝƽƣŝŤŮŜƎ0ŜƐűŘƏ¨ŔǁƢƣƩƕƨƝRs ŲfţƐūƱƽƺDŽƦŹŜŵƎefŲŗƏŕĂ3 40²ŲŹŢŸOōŹŵŘŕ ǃnŝkţş!ŝÀĖ0rŲŗƏųŞĂ34ƻƪƿŹǁ ƢƣƩƕƨƝRsƻƪƿƌƎƇţƍŶefŤűŘƏŕ ǃĂ340²Ź3ůŸƝƽƣŝŗƏYHŶƇ Ś Ə ŜƍŬƌŕ 45
46. 4.4.1 Using Bayes’ Theorem for Classification 340²ƀƵƖƤŸfØƓ ř
47. 4.4.1 Using Bayes’ Theorem for Classification ĘÍƪDŽƦƓ 9 (9 >2) %ŸƝƽƣŶ0ŠūŘųŦƏƌ ůƄƎŔĥáŵâá\¢$Ź9%Ÿ&ƓųƏƌ :; ƓĘÍƪDŽƦŝƝƽƣ<ŲŗƏèÖųŦƏƌ =;(!) ≡ Pr(! = A|$ = <)ųŦƏƌ ŢƐŹƝƽƣ<ŶśŠƏ!ŸixĽ¢Ŭƌ ůƄƎŔƝƽƣ<ŶśŘű! ≈ AŶŵƏèÖŝŒŘ¨ =;(!)Ź_Şŵ&ƓųƏƌ 47
48. 4.4.1 Using Bayes’ Theorem for Classification ŢŢŲŔƵƖƤŸfØƓ řƌ ŢƐŜƍŹD; ! = Pr $ = < ! ųĒŦƌ yƓĕƐźƒŜƏ»Ŷ:; ų=;(A)ƓfŤūŘƌ ŦƏųD; ! ŝƒŜƏŜƍ :; ƓfŦƏŢųŹö<Ŭƌ <øŶƝƽƣŸ7HŬƌ =;(A)ƓfŦƏŢųŹƍŜŸixĽ¢ƓfŤŵŘ ųņŤŘƌ ũŸ¦ÈŶůŘűŸƥƝơƼǂŲġ§ŤűŘşƌ 48
49. 4.4.2 Linear Discriminant Analysis for D = 1 Ă340²(D = 1)
50. 4.4.2 Linear Discriminant Analysis for D = 1 ƄŧŹD = 1ųfŦƏƌ ůƄƎġ§\¢Ź
    íŬƌ =;(A)ƓÀĖ0rųfŦƏƌ ţƍŶŔE" # = E# # =, …, = EG # ≡ E#ųfŦƏƌ ,űŸƝƽƣŲ0¡ŝòŤŘųŘřŢųŬƌ 50
51. ůŸyƓHƒŨƏų ŢřŵƏƌ 4.4.2 Linear Discriminant Analysis for D = 1 51
52. 4.4.2 Linear Discriminant Analysis for D = 1 j¢ƓųŮű<ŶĽŦƏķ0ŬŠćŚƏų ŢřŵƏƌ AŝÆƄŮū¨ŶƝƽƣŝ<ŬŮūYHH; Ÿ&Ź_Şş ŵƏƌ ŭŵƅŶH; ŝAŸĂŵĽ¢ŬŜƍĂ340²Ůű Ěřƌ 52
53. !Śź9 = 2Ų:" = :# ųŦƏƌ ŢŸųŞŔ AŝƝƽƣ1ŲŗƏųŞH" > H# ųŵƏƌ ůƄƎ ųŵƏƌ Ɲƽƣ2ŸųŞƇI»Ŭƌ ZÛĂŹH" = H# ƓAŶůŘűęŘű ųÅƄƏƌ 4.4.2 Linear Discriminant Analysis for D = 1 53
54. !ƓŸTŶéŦƌ J" = −1.25 , J# = 1.25 , E" # = E" # = 1 ųŤūƌ Ƅū:" = :# = 0.5ųŦƏųT ŸœŸçĂ(ÆfZÛ) ŝzŠƏƌ 4.4.2 Linear Discriminant Analysis for D = 1 54
55. FŸTŹ20%ŸĘÍƪDŽƦŸƲƣƫƞƽƹŬƌ ŢŸƌřŶƱƽƺDŽƦŝ,ű0ŜŮűŘƏYHŹƵƖƤ 0ŐQƓö<ŶěóŦƏŢųŝŲŞƏƌ „ƍƐūƪDŽƦŜƍĂ340²ŶƌƏÆfZÛƓœ ŸgĂŲéŦƌ 4.4.2 Linear Discriminant Analysis for D = 1 55
56. gŃŶŹ!ŝÀĖ0rŶƒřų0ŜŮűŘūųŤűƇ J", … , JG, :", … , :G, E#ƓfŦƏ…ĔŝŗƏƌ Ă340²(LDA)ŹƱƽƺDŽƦŸf&ƓţŮŞŸy ŶĴÚŤűƵƖƤ0ŐQƓƏƌ Ÿf&ŝƌşÚŘƍƐƏƌ 4.4.2 Linear Discriminant Analysis for D = 1 56
57. NŹ,ĘÍ&Ÿ¢Ŕ N; ŹƝƽƣ<ŸĘÍ&Ÿ¢ ŸƓfŦƏ…ĔŝŗƏƌ J; ŹƝƽƣ<Ÿ<øŵuVŬƌ E#ŹGƝƽƣŸ¼°0¡Ÿ8ĹuVųƅƐƏƌ :; ŹƪDŽƦŸ7HŲŘŘƌ ůƄƎ: O; = N;/NŬƌ 4.4.2 Linear Discriminant Analysis for D = 1 57
58. ąƎĮŦŠŴLDAŹĘÍ&ŝƝƽƣU¯ŸuV&Ɠ ›ŮűŘűŔ-ıŸ0¡Ɠ›ůÀĖ0rŶƒřųf ŦƏƌ ũŤűŢƐƍŸƱƽƺDŽƦŸf&ƓƵƖƤ0ŐQŶĴ ÚŦƏƌ 4.4.4ôŲŹƝƽƣũƐŪƐŝU¯Ÿ0¡E; #Ɠ›ůųŘ řţƍŶ{ŘfƓśşƌ 4.4.2 Linear Discriminant Analysis for D = 1 58
59. 4.4.3 Linear Discriminant Analysis for D > 1 Ă340²(D > 1)
60. 4.4.3 Linear Discriminant Analysis for D > 1 LDAƓē¢Ÿġ§\¢ŸYHŶš|ŦƏƌ ũŸūƆŶŔĘÍ&ŹŔGƝƽƣŝU¯ŸuVƵƝƫƿ ųŔ,Ɲƽƣ-ıŸ0¡-0¡đ2Ɠ›ů^\ĺÀĖ 0rŶƒřųfŦƏƌ ^\ĺÀĖ0rŹũƐŪƐŸ\¢ŝ1¾*ŸÀĖ0 rŶƒřųfŤűŔţƍŶũƐŪƐŸ\¢ŹäĽĽ$ Ɠ›ůƌ 60
61. 4.4.3 Linear Discriminant Analysis for D > 1 D = 2Ÿ^\ĺÀĖ0rŸixĽ¢ŸƞƽƳƓéŦƌ ŴŮŭŸTƇx" ĨƄūŹx# ĨŶũŮűTƓ1¤ŦƏų¤ ʼnŹ1¾*ŸÀĖ0rŶŵƏƌ pŹVar !" = Var !# ŲCor !", !# = 0Ŭƌ FŹ!" ų!# Ŷ0.7ŸäĽŝŗƏƌ 61
62. 4.4.3 Linear Discriminant Analysis for D > 1 ^\ĺÀĖ0rŸixĽ¢Ź ŢƔŵŬƌ !~V(J, Σ)Ůű¬şƌ E ! = JŹD¾*ŸuVƵƝƫƿŬƌ Cov ! = ΣŹD×DŸ0¡-0¡đ2Ŭƌ 62
63. =;(! = A)Ɠ Ŷ+ŤűŔţŮŞų Iť»Ŷ¢ěóƓđřų ųŵƏƌ D = 1ŸųŞŹŢƔŵŬŮūŷ 4.4.3 Linear Discriminant Analysis for D > 1 63
64. !ƓéŦƌ ůŸ¸.ŹũƐŪƐŸ¯ŽèÖ95%ŸŌWŬƌ çĂŹƵƖƤŸÆfZÛŬƌ GƝƽƣŜƍŸ20%ŸĘÍ&ųŔũƐŜƍLDAŲěóţ ƐūÆfZÛ(œŸgĂ)ƓFŸTŶéŦƌ 4.4.3 Linear Discriminant Analysis for D > 1 64
65. ƵƖƤŸÆfZÛŹH; A = HZ(A)ųŵƏAŸńHƓĒ ŤűŘƏƌ ůƄƎ ƓÎūŦƌ(< ≠ \) 4.4.3 Linear Discriminant Analysis for D > 1 65
66. ţŮŞŸƝǀƢƨƫƛDŽƬŸƪDŽƦŶLDAƓĴÚŦƏƌ ġ§\¢ŹbalanceųstudentŬƌ 10000%ŸĜăƪDŽƦƓÚŘūƌ ĜăƪDŽƦŸƘƽDŽÖŹ2.75%ŬŮūƌDž ŒŘ÷xŶĕŚƏŠŴůɎŝ…ĔŬƌ 4.4.3 Linear Discriminant Analysis for D > 1 66
67. żųůƆ ĜăƪDŽƦŸƘƽDŽÖŹŔgŃŸƪDŽƦŸƘƽDŽÖƌƎ ƇŘ&ŶŵƏŸŹ~ÑŬƌ ůƄƎŔ¥ŤŘƪDŽƦƓ›ŮűŞűŢŸ0ŐQŶıŦų ƘƽDŽÖŝŝƏŢųŝ ŲŞƏƌ ĜăƪDŽƦŲřƄşŘş»ŵƱƽƺDŽƦƓfŤūŜƍŷ ŢƐŹD ųNŸÃŝ_ŞŘųŔƌƎŏďŬƌ RŸYHŹD = 2ŲN = 10000ŬŜƍƂųƔŴ€Ŋ ŹŵŘų‡řŠŴ 4.4.3 Linear Discriminant Analysis for D > 1 67
68. žūůƆ ĜăƪDŽƦŸřŭdefaultŸ7HŹ3.33%ŬŮūƌ ůƄƎŔþjdefaultŶŵƍŵŘų ÍŦƏ0ŐQŲƇ 3.33%ŸƘƽDŽÖŶŵƏƌ LDAŸ0ŐQƌƎlŤŬŠŒŘƘƽDŽÖŶŵƏƌ 4.4.3 Linear Discriminant Analysis for D > 1 68
69. ƘƽDŽŶŹíŐŗƏƌ defaultŦƏŶdefaultŤŵŘų34ŦƏƘƽDŽųŔ defaultŤŵŘŶdefaultŦƏų34ŦƏƘƽDŽ ƘƽDŽŸíŐŶůŘűćŚƏŢųŹ_ Ŭƌ ŸƌřŵĒ(ËIđ2)ŝ#5Ŭƌ 4.4.3 Linear Discriminant Analysis for D > 1 69
70. ĒŶƌƏųLDAŹ104ŝdefaultŦƏų Ťūƌ gŃŶŹ81ŝdefaultŤű23ŝdefaultŤŵŜŮūƌ ůƄƎdefaultŤŵŜŮū9667Ÿřŭ23ŝļIJŚūƌ ÷xŒŘ»ŶƅŚƏƌŷ 4.4.3 Linear Discriminant Analysis for D > 1 70
71. ŬŠŴdefaultŤū333Ÿřŭ252ƇĕįŤūƌDž ,áŵƘƽDŽÖŹŘ»ŶĕŚƏŠŴdefaultŤū Ÿ ŸƘƽDŽÖŹŒŘƌ ƾƣƝŝŒŘƓÔfŤƌřųŤűŘƏƛDŽƬêŸĘ ÏŜƍŦƏųŔ252/333=75.7%ųŘř¢bŹDŠ+Ɛƍ ƐŵŘEĉ‰ŝŗƏƌ 4.4.3 Linear Discriminant Analysis for D > 1 71
72. 0ŐŸ‰ĉŹĐØdųŜÙÓdŲƇĹĔŬƌ xųÔÞxųŘřÚğŝŗƏƌ xŹßŠŝŗƏųŞŶ·µŲÀŤşł‰ŶŵƏÖŲŗ ƎŔŢŸYHŹ24.3%ųŘ ÔÞxŹßŠŝŵŘųŞŶ·µŲÀŤşŀ‰ŶŵƏÖ ŲŗƎŔŢŸYHŹ99.8%ųŒŘ 4.4.3 Linear Discriminant Analysis for D > 1 72
73. ŵƔŲŢƔŵŶxŝŘŸŜLJ ƵƖƤ0ŐQŹ,űŸ0ŐQŸ Ų­ƇĠqÖŝ Řƌ(ƜƗƣƻƪƿŝÀŤŘYH) ŢƐŹŔƘƽDŽŝŴŸƝƽƣŜƍ±ūŜŶŜŜƒƍŧŔ ũŸƘƽDŽŸÿNƓ­kĿŶ˜ŚƏųŘřŽKŬƌ
    ¦ƛDŽƬêŹgŃŶŹdefaultųŵƏƓļIJŮű 0ŐŦƏŢųŹĶŠūŘŜƇŤƐŵŘƌ ũřŘŮūYHƛDŽƬêŸƮDŽƤŶj†ŤűLDAƓ\ «ŤūƎŦƏƌ 4.4.3 Linear Discriminant Analysis for D > 1 73
74. ƵƖƤ0ŐQŹ ‚èÖD;(!)ŝ­Ƈ_ŞşŵƏƝƽƣ ŶĘÍ&Ɠ7Ǝ~űƏƌ !Śź2ƝƽƣŶ0ŠƏdefaultŸYH Ÿ¨ŹŔĘÍ&Źdefault=YesŶ0ŐţƐƏƌ ůƄƎŔ ‚èÖŶjŤű50%Ÿľ&Ų0ŐŤűŘƏų ĚŚƏŷ defaultŲŗƏŸĠ·æÖƓšūŘYHŹŔŢŸľ &ƓšƏųčŘŷ 4.4.3 Linear Discriminant Analysis for D > 1 74
75. ľ&Ɠ20%ŶšūYHŸû³ƓĒŶéŦƌ defaultŤū333ŸřŭĕįŤūŸŹ138(41.4%) ŬŮūƌ ţŮŞŸ50%Ÿľ&Ÿ¨Ź75.7%ŬŮūƌŷ ŧŘſƔŸPŤūŷ 4.4.3 Linear Discriminant Analysis for D > 1 75
76. ŬŠŴƇŭƑƔčŘŢųźŜƎťƈŵŘƌ ,ŸƘƽDŽÖųŤűŹ2.75%Ŝƍ3.73%ŶŝŮūƌ ƝǀƢƨƫêŜƍŦƏųľ&20%Ÿ¦ŝčŘŜƇŤƐ ŵŘƌŷ 4.4.3 Linear Discriminant Analysis for D > 1 76
77. ľ&Ɠ\«ŤūYHŸƘƽDŽÖŸƞƽƳƓīŨƏƌ ½ĨŝţŮŞŸľ&Ŭƌ ŘƑŘƑŵƘƽDŽÖŝľ&ŸĽ¢ųŤűƴǁƨƫţƐű ŘƏƌ 4.4.3 Linear Discriminant Analysis for D > 1 77
78. œŘgĂŝ,ŸƘƽDŽÖŬƌ ŇŘçĂŝdefaultŦƏŶdefaultŤŵŘų34ŤūƘ ƽDŽÖŬƌ ƚǀǂƢŸÏĂŹdefaultŤŵŘŶdefaultŦƏų34 ŤūƘƽDŽÖŬƌ 4.4.3 Linear Discriminant Analysis for D > 1 78
79. ľ&ŝ0.5Ÿ¨ŝ,ŸƘƽDŽÖŹ
    ÝŘƌ ŬŠŴŔŴŸƘƽDŽÖƓšūŘŜŶƌŮűľ&Ź\ :ŦƏƌ ƇŭƑƔŔŶƇ»ŖŵŒXŜƍľ&ƓÆfŦƏƌ 4.4.3 Linear Discriminant Analysis for D > 1 79
80. ROCªĂƓéŦƌ ĄĨŹdefaultŦƏŶdefaultŦƏų34ŤūÖŬƌ ůƄƎŔxŸŢųŬƌ 4.4.3 Linear Discriminant Analysis for D > 1 ½ĨŹdefaultŤŵŘŶdefault ŦƏų34ŤūÖŬƌ ůƄƎŔ1-ÔÞxŬƌ ŗƍƊƏľ&ŲŔůŸÖƓě óŤűƴǁƨƫŤūŸŝROCªĂ Ŭƌ 80
81. 4.4.3 Linear Discriminant Analysis for D > 1 81
82. 4.4.3 Linear Discriminant Analysis for D > 1 ŇĂŸŸʼnîŝ_ŞŠƐźŔčŘ0ŐŝŲŞƏƌ ũŸʼnîŹAUCųLźƐƏƌ RŸYHAUCŹ0.95ųňtŶŒŘƌ £ƆŸçĂŹŔŗƍƊƏľ&Ŷ śŘűůŸÖŝòŤŘĂŬƌ 6ƶDŽƢŸTŲŘřųŔ0rŝ ĹŵŮűŘƏųŞųŜŶŵƏƌ ŢŸYH34Ÿû³ŹŸŒ XƇŚŵŘƌ 82
83. 4.4.3 Linear Discriminant Analysis for D > 1
    ČáŶ0ŐŤūû³ŹĒŸƌřŶŵƏƌ “dž”Ɠ·/ŤūŘƌŔjïġŬƌ “DŽ”ũŸAjŬƌŔsÐġŬƌ defaultƪDŽƦŸYHŹ”dž”ŝdefault=YesŬƌ 83
84. 4.4.3 Linear Discriminant Analysis for D > 1 ĹĔŵýěĺƓŸĒŶéŦƌ False Pos. rateųTrue Pos. rateŸ0ÂŹGƝƽƣŸgŃ Ÿ%¢Ŭƌ Pos. pred. valueųNeg. Pred. valueŸ0ÂŹGƝƽƣŸ ÍŤū%¢Ŭƌ 84
85. 4.4.4 Quadratic Discriminant Analysis ¾340²
86. 4.4.4 Quadratic Discriminant Analysis LDAŲŹŔĘÍ&ŹŔGƝƽƣŝU¯ŸuVƵƝƫƿ ųŔ,Ɲƽƣ-ıŸ0¡-0¡đ2Ɠ›ů^\ĺÀ Ė0rŶƒřųfŤūƌ ¾340²(QDA)ŲŹŔĘÍ&ŝGƝƽƣU¯Ÿ uVƵƝƫƿų0¡-0¡đ2Ɠ›ů^\ĺÀĖ0 rŶƒřųfŦƏƌ IJŘŹ0¡-0¡đ2ŝ-ı(LDA)ŜGƝƽƣU¯ (QDA)ŜŬŷ yŲ¬şųŢřŵƏƌ !~V(J;, Σ;) Σ; ŝƝƽƣ<Ÿ0¡-0¡đ2Ŭƌ 86
87. 4.4.4 Quadratic Discriminant Analysis ŢŸfŸųŞŘůƇŸH; A Ź ųŵƏƌ QDAŹŔĘÍƪDŽƦ! = AŹŔH; A ŝ­Ƈ_ŞşŵƏ Ɲƽƣ<Ŷ0ŐŦƏƌ J;, Σ;, :; Źf&ƓÚŘƏƌ LDAųŹIJŮűQDAŹAŸ¾yŶŵŮűƏƌ 87
88. 4.4.4 Quadratic Discriminant Analysis QDAŸ¦ŝ}ũřŬŠŴLDAŸĻ”ŹŗƏŸLJ DíŸġ§\¢ŝŗƏYHŔLDAŹD(D + 1)/2%Ÿ ƱƽƺDŽƦƓfŦƏ…ĔŝŗƏƌ
    ¦QDAŹ9D(D + 1)/2%ƇŗƏƌ ŬŠŴŔQDAŹLDAƌƎƇ´ħ‰ŝŒŘƌ ƺƾƨƫƪƺƾƨƫŹũƐŪƐŗƏƌ 88
89. 4.4.4 Quadratic Discriminant Analysis ùĎŸçĂŹƵƖƤ0ŐQŶƌƏÆfZÛŔœĎŸ ÏĂŹLDAŸÆfZÛŔĀĎŸgĂŹQDAŸÆfZ ÛŬƌ pŹΣ" = Σ# ŔFŹΣ" ≠ Σ# Ŭƌ 89
90. 4.4.4 Quadratic Discriminant Analysis pŸ¨ŹLDAŸ¦ŝčţũř FŸ¨ŹQDAŸ¦ŝčţũř 90
91. 4.5 A Comparison of Classification Method 34•ÈŸÃĪ
92. 4.5 A Comparison of Classification Method ŢŸðŲŹǁƢƣƩƕƨƝRsŔLDAŔQDAŶůŘűġ §Ťūƌ 2ðŲŹkĭ'È(KNN)ŶůŘűġ§Ťūƌ ũƐŪƐŸƺƾƨƫƪƺƾƨƫŶůŘűġ§ŤűŘşƌ 92
93. 4.5 A Comparison of Classification Method ƄŧŹŔǁƢƣƩƕƨƝRsųLDAŸĽ$ŶůŘűġ §ŦƏƌ ġ§\¢ŝ1ů(D = 1)Ų2ůŸƝƽƣŝŗƏųŞŔ ŢŸyŜƍLDAŸj¢ƚƨƤŹ ų¬ŠƏƌ ^. ų^" ŹŔJ" ųJ# ųE#ŸĽ¢Ŭƌ 93
94. 4.5 A Comparison of Classification Method ǁƢƣƩƕƨƝRsŸj¢ƚƨƤŹ ŢƔŵŬŮūƌŷ ŴŭƍƇAŶjŤűĂŵĽ¢Ŭƌ ůƄƎŔǁƢƣƩƕƨƝRsƇLDAƇÆfZÛŹãĂŲ
    āŵƔŬƌ ůŸIJŘŹ­mÈƓ řŜŔÀĖ0rƓfŤű uVų0¡ƓěóŦƏŜŬƌ ŭŵƅŶD > 1ŲƇI»Ŭƌ 94
95. 4.5 A Comparison of Classification Method ûnůŸ•ÈŹŔIťû³ƓƇūƍŦŮű‡řŜƇ ŤƐŵŘ ŠŴ_ŹIJřû³ŶŵƏƌ LDAŹŔĘÍ&Ź,Ɲƽƣ-ıŸ0¡-0¡đ2Ɠ ›ůÀĖ0rŶƒřųfŤűŘūŷ ŬŜƍŔũŸfŝ’ƎïůųŞŹLDAŹǁƢƣƩƕƨ ƝRsƌƎ)ëŬƌ İŶŢŸfŝ’ƎïūŵŘųŞŹǁƢƣƩƕƨƝR sŸ¦ŝ)ëŬƌ 95
96. 4.5 A Comparison of Classification Method KNNŹŢŸðŲġ§Ťū¦ÈųŹ,şIJŮūƌŷ ĘÍ&! = AŸƝƽƣŹŔ !ŜƍĭŘŋŶ9%ŸĘÍ &Ɠ@ÒŤűƝƽƣƓ34ŦƏƌ ůƄƎŔKNNŹƯǂƱƽƺƫƾƨƝŵ•ÈŬƌ ŬŜƍŔZÛŝ,ÑĂťƈŵŘųŞŹŔLDAƉǁƢ ƣƩƕƨƝƌƎ)ëŬƌ ŬŠŴŔKNNŹŴŸġ§\¢ŝĹĔŜųŜŹ0Ŝƍ ŵŘƌ 96
97. 4.5 A Comparison of Classification Method QDAŹŔKNNųLDAśƌŽǁƢƣƩƕƨƝRsŸa; ÏƅūŘŵƇŸŬƌ QDAŹŔÆfZÛŝ2¾Ľ¢ŬŜƍĂŵ¦ÈƌƎƇ ´ħ‰ŹŒŘƌ ŲƇŔKNNƂŴŲŹŵŘƌ 97
98. 4.5 A Comparison of Classification Method 4ůŸ•ÈƓŔ6íŸơƭƾƚŸƪDŽƦƓŮűĴÚŤ űƅūƌ 3ůŹĂŵZÛŲŔÁƎ3ůŹňĂŵZÛŬƌ ũƐŪƐŶjŤűŔ100%ŸƪDŽƦƥƨƫƓŮūƌ ũŤűŔƘƽDŽÖƓěóŤūƌ KNNŹ9 = 1ŸųŞųCV(cross-validation)ųŘř•Èų 2íŐŤūƌ CVŶůŘűŹ5ðŲġ§ŦƏƌ ũƐŪƐŸơƭƾƚŹD = 2Ŭƌ ůƄƎġ§\¢ŹůŬƌ 98
99. 4.5 A Comparison of Classification Method Ÿůŝ ĂŵZÛ ŲŔŸů ŝňĂŵ ZÛŬƌ 99
100. 4.5 A Comparison of Classification Method ơƭƾƚ 1 2ůŸƝƽƣŲũƐŪƐĘÍ&Ź20 %Ŭƌ GƝƽƣŸĘÍ&ŹuV&ŝÞŵ ƏÐäĽŵÀĖ ¢Ŭƌ 100
101. 4.5 A Comparison of Classification Method ơƭƾƚ 1 LDAŝ
     ÝřƄşŘŮūƌ KNNŹưƽůŘūƌ QDAŹ´ħ‰ŝ…ĔŶŒŘŜ ƍLDAƌƎ‹:Ťūƌ ǁƢƣƩƕƨƝRsŹÀĖ0rŸ fƓśŜŵŘ0ŔLDAƌƎƂƔŸ ŭƋŮų9Ůūƌ 101
102. 4.5 A Comparison of Classification Method ơƭƾƚ 2 ĝfŹơƭƾƚ1ųƂƃ
     āŬƌ ūŬŤŔ2ůŸġ§\¢ŸļŶ-0.5 ŸäĽŝŗƏƌ ơƭƾƚ1ųäjáŵƱƳƙDŽƷǂ ƣŹƂųƔŴ
     āŬƌ 102
103. 4.5 A Comparison of Classification Method ơƭƾƚ 3 !" ų!# Źt0rŶƒřƌ GƝƽƣ50%ŸƪDŽƦŝŗƏƌ t0rŸÕŹƂųƔŴÀĖ0rų
     āŬŠŴŔuV&ŜƍijŘ&ŝ ^şŵƏ(JŶŗƏƌ ZÛĂŹãĂŲŔǁƢƣƩƕƨƝR sŝĴŤűŘūƌ LDAŹfŝÀĖ0rŬŜƍŷ 103
104. 4.5 A Comparison of Classification Method ơƭƾƚ 3 LDAƇǁƢƣƩƕƨƝRsƇŸ¦ ÈƌƎ)ƐűŘūƌ QDAŹÀĖ0rťƈŵŜŮūūƆ ŶŜŵƎ‹:Ťūƌ 104
105. 4.5 A Comparison of Classification Method ơƭƾƚ 4 Ɲƽƣ1Ÿġ§\¢ļŸäĽŹ0.5 ŲŔƝƽƣ2Ÿġ§\¢ļŸäĽŹ -0.5Ŭƌ ŢƐŹQDAŸfŶj†ŤűŘƏ ƌ ZÛŹ¾ªĂŶŵƏƌ ūŤŜŶQDAŝŸ•ÈƌƎ)Ɛ űŘƏŢųŝĕűCƐƏŷ 105
106. 4.5 A Comparison of Classification Method ơƭƾƚ 5 GƝƽƣŸĘÍ&ŹŔġ§\¢ļ ŶäĽŸŵŘÀĖ ¢Ŭƌ âá\¢ŹŔ!" #ų!# #ų!"×!# Ɠġ §\¢ųŤűŔǁƢƣƩƕƨƝĽ¢ ŜƍƠǂƴƿŤūƌ ũŸû³ŔZÛŹ¾ŶŵƏƌ QDAŝ
     ÝŘŘŷ ¾ŶKNN-CVŬƌ Ăŵ•ÈŹ‰ĉŝ‹Řƌ 106
107. 4.5 A Comparison of Classification Method ơƭƾƚ 6 ĞúŹơƭƾƚ5ų
     āŬŠŴŔâ á\¢ŹƇŮųēŅŵňĂĽ¢ ŜƍƠǂƴƾǂƞŤūƌ ũŸû³ŔQDAŲƇĴ1ŵƻƪƿ :ŝŲŞŵŜŮūƌ Ăŵ•ÈƌƎQDAŸ¦ŝƷơŬ ŠŴŔƌƎ´ħŵKNN-CVŝ
     Ýč ŜŮūƌ 107
108. 4.5 A Comparison of Classification Method ơƭƾƚ 6 KNN-1ŝ
     ݋ŜŮūƌ ƪDŽƦŝēŅŵňĂĽ¢ŲŗŮ űƇŔKŸ&ŝÀŤşĵ™ţƐŵŘų "ÑųŤűĤ{ŲŗƏŢųƓéŤű ŘƏƌ 108
109. 4.5 A Comparison of Classification Method Ÿ6ůŸ!ŹŔŴƔŵÕÇŲƇ)ëųŘř•ÈŹ ŵŘŢųƓéŤűŘƏƌ ũƐŪƐŶŘŘųŢŝŗƏƌ ÆfZÛŝĂŵųŞŹLDAųǁƢƣƩƕƨƝRsŝ ŘŘƌ ĴxŶňĂŵųŞŹQDAŝŘŘƌ ēŅŵňĂŸųŞŹKNNŝŘŘƌ ŬŠŴKŸ&Ź‘ĹŶĵſƁŞŬƌ 109