Sparkで生存時間解析をしたかった話

第1回 Machine Learning 15minutes! ( http://machine-learning15minutes.connpass.com/event/32889/ ) で発表した時の資料です。

Transcript

1. SparkͰੜଘ࣌ؒղੳΛ ͔ͨͬͨ͠࿩ ૣ઒ɹರ࢜ גࣜձࣾϦΫϧʔτίϛϡχέʔγϣϯζ
2. ࣗݾ঺հ • ૣ઒ ರ࢜ • ػցֶश΍σʔλ෼ੳΛͯ͠·͢ • גࣜձࣾϦΫϧʔτίϛϡχέʔγϣϯζʹ৽ଔೖࣾͯ͠ 2 ೥໨ • ίϛϡχςΟ׆ಈ: Japan.R ओ࠵ • ࣥච׆ಈ • σʔλαΠΤϯςΟετཆ੒ಡຊ, ʮPython ʹΑΔػցֶशʯΛ୲౰ • झຯ • Ϛϥιϯ: ϑϧϚϥιϯͷλΠϜ͸ 5 ࣌ؒ 4 ෼Ͱͨ͠ orz • ࣗసं: ୆࿷Ұप, ౦ژ͔Βവؗ·Ͱ૸ഁ, ശࠜͷࢁӽ͑ • ϑοταϧ: ٳΈͷ೔ʹͨ·ʹ, ༠͍ͬͯͩ͘͞ʂ ʂ • ిࢠ޻࡞: όονॲཧ͕མͪΔͱճస౮͕ޫΔΨδΣ οτΛ࡞ͬͯΈͨ • ՖՐ: ֶੜ࣌୅ͷαʔΫϧͰ೥ 1 ճͷଧͪ༲͛ 2/ 18
3. ͢Δ࿩ɺ͠ͳ͍࿩ • ͢Δ࿩ • Spark Ͱੜଘ࣌ؒղੳΛࢼͯ͠஍ཕΛ౿Μͩ࿩ • Ϟσϧͷ؆୯ͳ঺հ • ͠ͳ͍࿩ • Spark, Scala ͷৄࡉͳղઆ • ొ৔͢Δίʔυ΁ͷποίϛ͸׻ܴʂ • ஍ཕΛղܾ͢Δํ๏ • Ϟσϧͷৄࡉͳ࿩ 3/ 18
4. Sparkͱ͸ • େن໛σʔλ޲͚ͷ෼ࢄॲཧϑϨʔϜϫʔΫ • Java, Scala, Python, R ͔Βར༻Մೳ ਤ 1: ׆ൃͳ։ൃ (https://en.wikipedia.org/wiki/Apache Spark ΑΓҾ༻) 4/ 18
5. ੜଘ࣌ؒղੳ • Πϕϯτ͕ى͖Δ·Ͱͷ࣌ؒͱɺͦͷΠϕϯτؒͷؔ܎Λ෼ੳ͢Δख๏ • ˓˓࣌ؒޙʹʷʷ͢Δ֬཰ΛϞσϦϯάͰ͖Δ • ϞσϦϯάྫ • ফඅऀ͕঎඼Λߪೖ͢Δִؒ • ػցͷނো༧ଌ • පؾͷ࠶ൃ΍ࢮ๢ 5/ 18
6. ੜଘ࣌ؒղੳͷಛ௃ • ଧͪ੾Γͱ͍͏ߟ͑ํ͕ ͋Δ • ෼ੳର৅ͱ͢ΔΠϕϯτ ͕ى͖ΔલʹԿΒ͔ͷཧ ༝ʹΑͬͯ؍ଌ͕ग़དྷͳ ͔ͬͨ৔߹ ਤ 2: https://www1.doshisha.ac.jp/ mjin/R/36/36.html ΑΓҾ༻ 6/ 18
7. SparkͰར༻Մೳͳੜଘ࣌ؒղੳͷϞσϧ • AFT(Accelerated Failure Time) Ϟσϧ • ଧͪ੾Γσʔλ༻ͷύϥϝτϦοΫͳϞσϧ • Spark Ͱ͸ϫΠϒϧ෼෍ΛԾఆͨ͠ϫΠϒϧճؼ͕࣮૷ࡁΈ • ܗঢ়ύϥϝʔλڞ௨ mɺई౓ύϥϝʔλ η(X) ͕ڞมྔͰมΘΔ • F(t|X) = 1 − exp ( − ( t η(X) )m ) • η(X) = exp(βX) 7/ 18
8. αϯϓϧσʔλͷ࡞੒ αϯϓϧσʔλͷ࡞੒ > set.seed(71) > n <- 1000 > m <- 2 > beta < -1 > beta0 <- 10 > x <- rnorm(n) > data.df <- data.frame( > tt = rweibull(n, shape = m, > scale=exp(beta * x + beta0)), > x = x, status=rep(1,n)) 8/ 18
9. RݴޠʹΑΔϫΠϒϧճؼ R ݴޠʹΑΔϫΠϒϧճؼ > library(survival) > survreg(Surv(tt, status) ~ x, data=data.df, dist="weibull") Call: survreg(formula = Surv(tt, status) ~ x, data = data.df, dist = " weibull") Coefficients: (Intercept) x 10.0162268 -0.9916226 Scale= 0.491728 Loglik(model)= -10618.8 Loglik(intercept only)= -11366.3 Chisq= 1494.89 on 1 degrees of freedom, p= 0 n= 1000 9/ 18
10. Sparkͷ࣮ߦ؀ڥ Docker ্Ͱͷ࣮ߦ sudo docker run --name spark-notebook -p 9000:9000 andypetrella/ spark-notebook:0.6.3-scala-2.10.5-spark-1.6.1-hadoop-2.7.2- with-parquet Ͱ Spark Notebook Λىಈͯ͠࢖͍ͬͯ·͢ɻ 10/ 18
11. SparkͰͷσʔλಡΈࠐΈ1 Scala ͷίʔυ import org.apache.spark.sql.functions._ import org.apache.spark.mllib.linalg.{Vector, Vectors} val sqlContext = new org.apache.spark.sql.SQLContext(sc) import sqlContext.implicits._ val df = sqlContext.read.json("sampledata.json") 11/ 18
12. SparkͰͷσʔλಡΈࠐΈ2 Scala ͷίʔυ (100 ݅ͷσʔλͰࢼ͢) case class MyDF(label: Double, censor: Double, features: org.apache.spark.mllib.linalg.Vector) val training = df. withColumn("x", df("x").cast("Double")). withColumn("tt", df("tt").cast("Double")). withColumn("status", df("status").cast("Double")). select("tt", "status", "x"). map(row => MyDF(row.getDouble(0), row.getDouble(1), Vectors.dense(row.getDouble(2)))). toDF("label", "censor", "features"). limit(100) 12/ 18
13. SparkͰͷϞσϧߏங Scala ͷίʔυ import org.apache.spark.ml.regression.AFTSurvivalRegression import org.apache.spark.mllib.linalg.Vectors val quantileProbabilities = Array(0.3, 0.6) val aft = new AFTSurvivalRegression() .setQuantileProbabilities(quantileProbabilities) .setQuantilesCol("quantiles") val model = aft.fit(training) println(s"Coefficients: ${model.coefficients} Intercept: " + s"${model.intercept} Scale: ${model.scale}") model.transform(training).show(false) 13/ 18
14. ݁Ռ ύϥϝʔλਪఆΛͯ͠ɺΠϕϯτͷൃੜΛ༧ଌ͍ͯ͠Δ Coefficients: [-0.94] Intercept: 9.94 Scale: 0.49 +---------+------+--------+-----------+--------------------+ |label |censor|features|prediction |quantiles | +---------+------+--------+-----------+--------------------+ |42227.15 |1.0 |[-0.43] |31156.57 |[18784.58,29848.00] | |31827.69 |1.0 |[-0.45] |31608.28 |[19056.92,30280.74] | |18740.14 |1.0 |[-0.48] |32552.67 |[19626.31,31185.47] | |9580.26 |1.0 |[0.42] |14050.37 |[8471.10,13460.26] | |9225.93 |1.0 |[-0.42] |30751.77 |[18540.52,29460.20] | 14/ 18
15. But 1000 ݅ͷσʔλΛೖΕΔͱɾ ɾ ɾ org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 235.0 failed 1 times, most recent failure: Lost task 0.0 in stage 235.0 (TID 127, localhost): java.lang.AssertionError: assertion failed: AFTAggregator loss sum is infinity. Error for unknown reason. ͱΤϥʔΛు͘ɻhttps://issues.apache.org/jira/browse/SPARK-13322 ͷνέο τͰ͸ɺڞมྔΛඪ४Խͨ͠ͳ͍৔߹ʹى͖Δͱ͋Δ͕ɺࠓճ͸ඪ४Խ͞Ε ͍ͯΔɻ 15/ 18
16. ·ͱΊ • ࣗલͰσʔλΛੜ੒ͯ͠ Spark Ͱੜଘ࣌ؒղੳΛͨ͠ • 100 ݅ͷΑ͏ʹখ͞ͳσʔλɾηοτͳΒ໰୊ͳ͍͕ɺ1000 ݅ʹͳΔͱ ΤϥʔΛు͘ • Spark ͷϝʔϦϯάϦετͰ࣭໰͍ͯ͠Δ࠷தͰ͢ • ੜଘ࣌ؒղੳΛ͢ΔͳΒ R ݴޠ͕ॆ࣮͍ͯ͠·͢ 16/ 18
17. ࢀߟࢿྉ • https://en.wikipedia.org/wiki/Apache Spark • https://www1.doshisha.ac.jp/ mjin/R/36/36.html • https://spark.apache.org/docs/1.6.0/ml-classification-regression.html • http://spark.apache.org/docs/latest/sql-programming-guide.html • http://aaaaushisan.blogspot.jp/2011/12/survreg-weibreg.html • http://blog.gepuro.net/archives/102 17/ 18
18. Enjoy Spark! 18/ 18