Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
spark.ml の API で XGBoost を扱いたい!#shokaispark
Search
KOMIYA Atsushi
May 11, 2016
Programming
3
4.4k
spark.ml の API で XGBoost を扱いたい!#shokaispark
『詳解 Apache Spark』出版記念イベントでの発表資料です。
http://connpass.com/event/30375/
KOMIYA Atsushi
May 11, 2016
Tweet
Share
More Decks by KOMIYA Atsushi
See All by KOMIYA Atsushi
#JJUG Java における乱数生成器とのつき合い方
komiya_atsushi
5
5k
#JJUG Fork/Join フレームワークを効率的に正しく使いたい
komiya_atsushi
0
430
[#JSUG] SmartNews における container friendly な Spring Boot アプリケーション開発
komiya_atsushi
1
11k
Java のデータ圧縮ライブラリを極める #jjug_ccc #ccc_c7
komiya_atsushi
4
4.4k
#devsumi 自然言語処理・機械学習によるファクトチェック業務の支援
komiya_atsushi
1
4.1k
SmartNews Ads における機械学習の活用とその運用 #mlops
komiya_atsushi
3
19k
GBDT によるクリック率予測を高速化したい #オレシカナイト vol.4
komiya_atsushi
5
1.2k
Maven central repository の artifact をランキングする #渋谷java
komiya_atsushi
0
1.2k
確率的データ構造を Java で扱いたい! #JJUG
komiya_atsushi
6
2.2k
Other Decks in Programming
See All in Programming
DynamoDB コスト最適化っぽいことの基本 with Terraform
kuro_kurorrr
2
250
Temporalを取り巻く仕様を整理する
sajikix
0
110
Findy - エンジニア向け会社紹介 / Findy Letter for Engineers
findyinc
2
81k
HMSコンペ 11th Solution (team : kansai-kaggler)
t88
1
680
Namespace on read
tagomoris
2
370
Harnessing Large Language Models for Training-free Video Anomaly Detection
tereka114
1
1.3k
入社1ヶ月でここまでやった!Findy Toolsインフラ支援の最適化
rvirus0817
6
1.4k
Introduction of Happy Eyeballs Version 2 (RFC8305) to the Socket library
coe401_
1
220
英語
s_shimotori
1
220
From Spring Boot 2 to Spring Boot 3 with Java 22 and Jakarta EE
ivargrimstad
0
1.9k
Android開発者のための Kotlin Multiplatform入門
ntaro
0
190
GraphQL はいいぞ! ~Laravel で学ぶ GraphQL 入門~
azuki
1
160
Featured
See All Featured
Building an army of robots
kneath
301
42k
Thoughts on Productivity
jonyablonski
64
4.1k
Raft: Consensus for Rubyists
vanstee
134
6.5k
How GitHub Uses GitHub to Build GitHub
holman
471
290k
RailsConf 2023
tenderlove
16
720
XXLCSS - How to scale CSS and keep your sanity
sugarenia
245
1.2M
Agile that works and the tools we love
rasmusluckow
325
20k
Reflections from 52 weeks, 52 projects
jeffersonlam
346
19k
The Pragmatic Product Professional
lauravandoore
29
6.1k
Happy Clients
brianwarren
94
6.5k
Being A Developer After 40
akosma
72
580k
Music & Morning Musume
bryan
43
5.9k
Transcript
spark.ml ͷ API Ͱ XGBoost Λѻ͍͍ͨʂ 2016-05-11 ʰৄղ Apache Sparkʱग़൛ه೦Πϕϯτ
KOMIYA Atsushi (@komiya_atsushi)
͓·͑ͩΕΑ
KOMIYA Atsushi @komiya_atsushi
Today’s topic
on
XGBoost • ޯϒʔεςΟϯάͷ࣮ͷҰͭ • ܾఆʹର͢ΔޯϒʔεςΟϯάɺ MLlib Ͱ GBTClassifier / GBTRegressor
ͱ ࣮ͯ͠͞Ε͍ͯΔ • ༧ଌਫ਼ͷߴ͞ͳͲ͔ΒɺKaggler ͳํʑΛ த৺ʹਓؾ͕͋ΔʢͬΆ͍ʣ
spark.ml ͷ API Ͱɺ XGBoost Λ Spark ্Ͱ ѻ͍͍ͨʂ
spark.ml ͷ API Ͱѻ͑Δͱ… • spark.ml ͕ఏڙ͢Δ֤छػೳΛ༗ޮ׆༻Ͱ͖Δ • ಛநग़ɾมɾબ •
ύϥϝʔλͷάϦουαʔν • ύΠϓϥΠϯ • ަࠩݕূ… ͳͲ
͜ͷൃදͰ͓͢Δ͜ͱ • XGBoost on Spark ͷݱঢ় • spark.ml ͷ API
ͰػցֶशΞϧΰϦζϜΛ ࣮͢ΔࡍͷϙΠϯτ • ಛʹΠϯλϑΣʔε෦ʹண͢Δ
XGBoost & Spark
XGBoost on Spark • Spark ্Ͱ XGBoost Λ͓͏ͱ͢Δͱɺ ݱঢ়Ͱબࢶ 2
ͭ • SparkXGBoost • xgboost4j-spark
SparkXGBoost • https://github.com/rotationsymmetry/sparkxgboost • XGBoost ͱಉ͡ޯϒʔεςΟϯάπϦʔΛɺSpark ͚ ʹ pure Scala
Ͱ࣮͍ͯ͠Δ • Spark packages ʹొ͞Ε͍ͯΔ • ΦϦδφϧͷ XGBoost ʹͲ͜·Ͱ࣮ͳ࣮ͳͷ͔ෆ໌ • ver 0.6 ·ͰͷϩʔυϚοϓ͕͋Δ͕ɺ։ൃ͕׆ൃͰͳ͍ • ࠷ޙͷίϛοτࡢ 11 ݄ɺver 0.2
xgboost4j-spark • DMLC ͕ఏڙ͢Δެࣜͷ Spark integration • ͨͩ͠ɺDataFrame ʹରԠ͍ͯ͠ͳ͍ •
XGBoost ຊମͷ git ϦϙδτϦ্Ͱϝϯς͞Ε͍ͯΔ • ֶश͓Αͼ༧ଌͷ۩ମతͳॲཧɺJNI ܦ༝Ͱ C++ ࣮ʹ͓ͤ • ֶश࣌ͷϫʔΧʔؒͷ௨৴ʹ Rabit Λར༻͍ͯ͠Δ • Maven central ʹొ͞Ε͍ͯͳ͍ • ར༻͢ΔʹྑϏϧυඞਢ
ࠓճ… • SparkXGBoost ͷΑ͏ʹɺXGBoost Λֶशث ؚΊͯ pure Scala Ͱ࠶࣮͢Δͷϋʔυϧ ͕ߴ͍
• xgboost4j-spark ͕ࢀর͢Δ xgboost4j Λ ϕʔεʹɺspark.ml ͷ API Ͱϥοϓͯ͠ΈΔ
spark.ml internals (ΏΔ;Θ)
spark.ml ͷ࣮ΛಡΉ • spark.ml ʹ͓͚ΔػցֶशΞϧΰϦζϜͷ ࣮͓࡞๏ΛΔʹͲ͏ͨ͠ΒΑ͍͔ʁ • MLlib ͕ఏڙ͢Δ֤छΞϧΰϦζϜͷ࣮Λ ಡΉͷ͕Ұ൪ͷۙಓ
spark.ml ͷ࣮ΛಡΉ • ࣮ΛಡΉͷʹ͓͢͢ΊͳػցֶशΞϧΰϦζϜ • ϩδεςΟοΫճؼ • LogisticRegression / LogisticRegressionModel
• ܾఆ (ྨ) • DecisionTreeClassifier / DecisionTreeClassificationModel • ܾఆ (ճؼ) • DecisionTreeRegressor / DecisionTreeRegressionModel
spark.ml ʹ͓͚Δػցֶशͷ࣮ • ػցֶशΞϧΰϦζϜͷֶशثɺΛḷΔͱ Estimator Ϋϥεʹߦ͖ண͘ • ֶशثʹΑͬͯಘΒΕΔ༧ଌϞσϧɺΛḷΔͱ Transformer Ϋϥεʹߦ͖ண͘
• ຊॻͷ pp.217-218 Λࢀর • ͨͩ͠ͲͪΒ Estimator Transformer Λ extends ͍ͯ͠ΔͱݶΒͳ͍
ֶशثͷΫϥε֊ &TUJNBUPS 1SFEJDUPS $MBTTJpFS 1SPCBCJMJTUJD$MBTTJpFS ճؼΞϧΰϦζϜͷଟ͘ 1SFEJDUPSΛFYUFOET͍ͯ͠Δ ྨΞϧΰϦζϜͷଟ͘ 1SPCBCJMJTUJD$MBTTJpFSΛFYUFOET͍ͯ͠Δ
༧ଌϞσϧͷΫϥε֊ 5SBOTGPSNFS 1SFEJDUJPO.PEFM $MBTTJpDBUJPO.PEFM 1SPCBCJMJTUJD$MBTTJpDBUJPO.PEFM 1SFEJDUPSʹରԠ͢Δ ༧ଌϞσϧͷΫϥεͱͳΔ 1SPCBCJMJTUJD$MBTTJpFSʹରԠ͢Δ ༧ଌϞσϧͷΫϥεͱͳΔ
ֶशثͱ༧ଌϞσϧͷ࣮
Predictor Ϋϥε • ΧϥϜ • label: ਖ਼ղϥϕϧΛ࣋ͭΧϥϜ • features: ಛϕΫτϧΛ࣋ͭΧϥϜ
• prediction: ༧ଌ͞Εͨϥϕϧ͕ઃఆ͞ΕΔΧϥϜ • ϝιου • train (நϝιου): ֶशॲཧΛ࣮͢Δ • extractLabeledPoints: DataFrame ͔Β RDD[LabeledPoint] Λੜͯ͘͠ΕΔϝιου
Classifier Ϋϥε • ΧϥϜ • rawPrediction: ༧ଌϞσϧ͕ੜͨ͠ੜͷ ͕ઃఆ͞ΕΔΧϥϜ • ༧ଌϥϕϧɺ͜ͷΛجʹٻΊΒΕΔ
ProbabilisticClassifier Ϋϥε • ΧϥϜ • probability: (ೋྨͰ͋Ε) ਖ਼ղϥϕϧ͕ 1 Ͱ͋Δͱ༧ଌ͞ΕΔ͕֬ઃఆ͞ΕΔΧϥϜ
• ύϥϝʔλ • threshold: ༧ଌ֬ (probability ΧϥϜ) ʹج͍ͮ ͯ 0/1 ʹৼΓ͚Δࡍͷ͖͍͠
PredictionModel Ϋϥε • ϝιου • transform: transformImpl ϝιουΛݺͼग़͚ͩ͢ • transformImpl:
༩͑ΒΕͨ DataFrame ͷͦΕͧΕ ͷߦ͝ͱʹ predict ϝιουΛݺͼग़͢ • predict (நϝιου): ༩͑ΒΕͨಛϕΫτϧ͔ Β༧ଌ݁ՌΛੜ͢ΔॲཧΛ࣮͢Δ
ClassificationModel Ϋϥε • ϝιου • transform: predict ϝιου predictRaw &
raw2Prediction ϝιουΛݺͼग़ͯ͠༧ଌ݁ՌΛٻΊΔ • predict: predictRaw ϝιουͷ݁ՌΛ raw2Prediction ʹ͠ ͯ༧ଌϥϕϧΛฦ͢ • predictRaw (நϝιου): ༧ଌϞσϧΛ༻͍ͯੜͷ༧ଌΛ ฦ͢ॲཧΛ࣮͢Δ • raw2Prediction (நϝιου): ༧ଌϞσϧ͕ੜͨ͠ੜͷ༧ ଌ͔ΒϥϕϧΛ༧ଌॲཧΛ࣮͢Δ
ProbabilisticClassificationModel Ϋϥε • ϝιου • predictRaw (நϝιου): ClassificationModel ʹಉ͡ •
raw2ProbabilityInPlace (நϝιου): ੜͷ༧ଌ͔Β༧ଌ ֬ʹม͢ΔॲཧΛ࣮͢Δ • predictProbability: predictRaw ϝιουͷ݁ՌΛ raw2ProbabilityInPlace ϝιουʹͯ͠༧ଌ֬ʹม͢Δ • probability2Prediction: ༧ଌ͔֬Β༧ଌϥϕϧΛฦ͢ • raw2Prediction: ੜͷ༧ଌ͔Β༧ଌϥϕϧΛฦ͢
ֶशثɾ༧ଌϞσϧͷ࣮ͷϙΠϯτ (1) • ྨΞϧΰϦζϜͱճؼΞϧΰϦζϜͰ࣮ΫϥεΛ ͚Α͏ • MLlib ͰɺϥϯμϜϑΥϨετޯϒʔεςΟ ϯάπϦʔͷΑ͏ʹɺྨʹճؼʹ͑ΔΞϧ ΰϦζϜͦΕͧΕͷ࣮Ϋϥε͕༻ҙ͞Ε͍ͯΔ
• e.g. GBTClassifier and GBTRegressor
ֶशثɾ༧ଌϞσϧͷ࣮ͷϙΠϯτ (2) • ྨΞϧΰϦζϜͷ࣮ • ֶशثͷ࣮Ϋϥε ProbabilisticClassifier Λ extends ͠Α͏
• ༧ଌϞσϧͷ࣮Ϋϥε ProbabilisticClassificationModel Λ extends ͠Α͏ • (ςϯϓϨతͳϝιουͷ࣮Λআ͚) predictRaw, raw2probabilityInPlace ϝιουΛ࣮͢Δ͚ͩͰࡁΉ
ֶशثɾ༧ଌϞσϧͷ࣮ͷϙΠϯτ (3) • ճؼΞϧΰϦζϜͷ࣮ • ֶशثͷ࣮Ϋϥε Predictor Λextends ͠Α͏ •
༧ଌϞσϧͷ࣮Ϋϥε PredictionModel Λ extends ͠Α͏ • predict ϝιουΛ࣮͢Δ͚ͩͰࡁΉ
ύϥϝʔλ
spark.ml ʹ͓͚Δύϥϝʔλ • ػցֶशʹϋΠύʔύϥϝʔλͷνϡʔχϯά͕ ͖ͭͷ • spark.ml ͰάϦουαʔνͷػೳΛఏڙ͍ͯ͠Δ • spark.ml
ͰػցֶशΞϧΰϦζϜΛ࣮͢Δࡍɺ ύϥϝʔλνϡʔχϯάͰ͖ΔΑ͏ߟྀ͕ඞཁ
ύϥϝʔλͷ࣮ྫ trait XGBoostGeneralParams extends Params { final val booster: Param[String]
= new Param(this, "booster", // ύϥϝʔλ໊ "which booster to use, can be gbtree or gblinear.", // આ໌ // ύϥϝʔλʹର͢ΔόϦσʔγϣϯϧʔϧ ParamValidators.inArray(Array("gbtree", "gblinear"))) // setter, getter Λ༻ҙ͢Δ def setBooster(value: String): this.type = set(booster, value) def getBooster: String = $(booster) // σϑΥϧτΛઃఆ͢Δ setDefault(booster, "gbtree") }
ύϥϝʔλͷ࣮ϙΠϯτ (1) • ύϥϝʔλΛఆٛ͠Α͏ • ܕ • Param, DoubleParam, IntParam,
FloatParam, LongParam… • ύϥϝʔλ໊ • આ໌ • όϦσʔγϣϯ • ParamValidators ͕ఏڙ͢ΔϑΝΫτϦϝιουΛར༻͢Δ
ύϥϝʔλͷ࣮ϙΠϯτ (2) • getter / setter Λ༻ҙ͠Α͏ • σϑΥϧτΛઃఆ͠Α͏ •
͜ͷ͋ͨΓςϯϓϨతͳ࣮ʹͳΔ
spark.ml-friendly XGBoost
xgboost-dataframe-prototype • https://github.com/komiya-atsushi/xgboost- dataframe-prototype • repo ໊ʹ͋Δͱ͓ΓɺϓϩτλΠϓͰ͢ • ͝ར༻͍ͨͩ͘ࡍ͝ҙΛ •
ֶश࣌ͷࢄॲཧ͍ͯ͠·ͤΜ • Rabit ͷ API ΛѲ͢Δඞཁ͕͋ΔͷͰ…
·ͱΊ
·ͱΊ • XGBoost Λࡐʹɺspark.ml ͷ API Ͱػցֶश ΞϧΰϦζϜΛ࣮͢ΔϙΠϯτΛ͓͠·ͨ͠ • ֶशثɾ༧ଌϞσϧͷΫϥε
• ύϥϝʔλ • Έͳ͞·ͷ Spark ্Ͱͷػցֶशͷ࣮ͷࢀߟ ʹͳΕ͍Ͱ͢
Thank you!