Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Sparkによる分散処理 / 2015-01-16 PyData.Tokyo#3
Search
shunsukeaihara
January 17, 2015
Technology
11
3.4k
Sparkによる分散処理 / 2015-01-16 PyData.Tokyo#3
shunsukeaihara
January 17, 2015
Tweet
Share
More Decks by shunsukeaihara
See All by shunsukeaihara
BONXを支える技術:発話区間検出(VAD)の話/Akerun & BONX Tech Talk
shunsukeaihara
4
7.4k
Goのnet.TCPConnの話/shibuya.go01
shunsukeaihara
2
720
Norikra in Gunosy Network Ads@Norikra meetup #2
shunsukeaihara
1
5.9k
LevelDB on S3 As A KVS
shunsukeaihara
1
2.6k
色恒常性仮説に基づく色補正ライブラリcolorcorrect / 2015-01-31-kantocv27
shunsukeaihara
3
2.2k
ゼロから始めた Gunosyアドサーバ開発運用記 / 2014-12-16-dots
shunsukeaihara
6
1.1k
Gunosy.Go#5 index/io/log
shunsukeaihara
0
140
Gunosy.go#2 package/compress
shunsukeaihara
0
93
Other Decks in Technology
See All in Technology
React Aria で実現する次世代のアクセシビリティ
ryo_manba
4
1k
PdMはどのように全てのスピードを上げられるか ~ 非連続進化のための具体的な取り組み ~
sansantech
PRO
1
140
デジタル化・DX推進あるある
y150saya
0
240
なにもしてないのにNew Relicのデータ転送量が増えていたときに確認したこと
tk3fftk
2
170
AIで変わるテスト自動化:最新ツールの多様なアプローチ/ 20240910 Takahiro Kaneyama
shift_evolve
0
190
疎通2024
sadnessojisan
5
1k
実践的なバグバウンティ入門
scgajge12
4
2.4k
App Router を実プロダクトで採用して見えてきた勘所をちょっとだけ紹介
marokanatani
0
760
RAGHack: Building RAG apps in Python
pamelafox
0
160
タイミーのBraze活用 ~PUSH通知を活用したレコメンド~
ozeshun
2
150
突撃! 隣のAmazon Bedrockユーザー 〜YouはどうしてAWSで?〜
minorun365
PRO
3
300
2024年版 運用者たちのLLM
nwiizo
3
510
Featured
See All Featured
Documentation Writing (for coders)
carmenintech
65
4.3k
Practical Orchestrator
shlominoach
185
10k
Web development in the modern age
philhawksworth
204
10k
Fashionably flexible responsive web design (full day workshop)
malarkey
401
65k
For a Future-Friendly Web
brad_frost
174
9.3k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
157
15k
The Power of CSS Pseudo Elements
geoffreycrofte
71
5.2k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
248
20k
Thoughts on Productivity
jonyablonski
66
4.2k
Designing for Performance
lara
604
68k
Principles of Awesome APIs and How to Build Them.
keavy
125
16k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
226
52k
Transcript
SparkʹΑΔࢄॲཧ (ͱPythonͰͷࢄॲཧ) Gunosy Inc. Shunsuke Aihara
ࣗݾհ • ҄൧ݪढ़հ (http://argmax.jp) @shunsukeaihara • GunosyͷϚωʔδϟʔ • ࠂ৴γεςϜͷ։ൃશମͱR&DܥΛ୲ •
ઐ: ܭࢉݴޠֶ • PythonͱඇಉظࢄγεςϜΛΉ • ը૾ॲཧɾԻ৴߸ॲཧͰ͍Ζ͍ΖϥΠϒϥϦ࡞ͬͯΔ • https://bitbucket.org/aihara
Agenda • Spark֓ཁ • ࢄॲཧ(ͱSpark)ͷ • GunosyͰͷSparkͷϢʔεέʔε • PythonͰͷࢄॲཧΤίγεςϜ
Sparkʹ͍ͭͯ(1) • HadoopͷΤίγεςϜ(HDFS, MESOS, YARN)ͱ࿈ܞ͢ΔΦϯϝϞ Ϧࢄॲཧܥ • Resillient Distributed Datasetsͱ͍͏োੑΛ࣋ͬͨࢄσʔλߏ
ʹର͢Δࢄϓϩάϥϛϯάڥ • RDDʹద༻͢ΔฒྻܭࢉΛɺߴ֊ؔͷνΣΠϯͷܗͰScalaɺ PythonͰ࣮ߦ • immutableͳσʔλߏ • RDDͷཁૉΫϥελͷΦϯϝϞϦʹࢄɾϨϓϦέʔγϣϯ • ഁଛɾϩετͨ͠σʔλӬଓԽͨ͠ݩσʔλ͔Β෮ݩ
Sparkʹ͍ͭͯ(2) • RDDʹର͢Δࢄॲཧج൫ͷ্ʹҎԼΛ࣮ • σʔλετϦʔϜॲཧ(Spark Streaming) • ࢄSQL(SparkSQL) • ࢄػցֶशϥΠϒϥϦ(Mllib)
• ࢄάϥϑॲཧϥΠϒϥϦ(GraphX)
ࢄॲཧ(ͱSpark)ͷ
େنσʔλࢄॲཧͷ؊ • ΫϥελϚωʔδϝϯτ • σʔλͷࢄஔͷࣗಈԽ • σʔλଟॏԽ/ฒྻReadʹΑΔߴԽ • σʔλϩʔΧϦςΟΛอͬͨܭࢉ •
োੑ / ࠶ૹɾ࠶ܭࢉॲཧ
HadoopʹࢸΔ·Ͱ • ෳࡶͳฒྻॲཧϝοηʔδύογϯάͰಠࣗʹ࣮͢Δͱେม • εέϧτϯฒྻϓϩάϥϛϯά(Cole, 1989) • සग़͢ΔฒྻܭࢉύλʔϯͷΈ߹ΘͤͰɺ༷ʑͳฒྻॲཧΛߏతʹߏங ͢ΔؔϓϩάϥϛϯάͷΈͱෳͷ࣮ •
σʔλฒྻεέϧτϯ(map, fold/reduce, filter, zip…) • σʔλͷҟͳΔ෦ʹɼಉ࣌ʹಉ͡ૢ࡞Λߦ͏ܭࢉύλʔϯ • λεΫฒྻεέϧτϯ(pipe, farm…) • σʔλͷετϦʔϜʹରͯ͠ɼͦΕͧΕܭࢉΛద༻ͨ͠σʔλετϦʔ ϜΛฦ͢ύλʔϯ
εέϧτϯฒྻϓϩάϥϛϯά މৼߐ ؠ࡚ӳ࠸ εέϧτϯฒྻϓϩάϥϛϯάใॲཧ 7PM /P QQ
HadoopҎલͷࢄॲཧ • MPI άϦουγΣϧΛ༻͍࣮ͯ • σʔλͷஔࣗͰϚωʔδ • ڞ༗ϝϞϦ͔ڞ༗FSʹࣗͰஔ͕લఏ • ڊେσʔλͷஔͱͯ໘
• োੑಠ࣮ࣗͰอূ • ϝϞϦʹࡌΓΒͳ͍σʔλΛѻ͏ͷ͍͠
T-shirts message@WOMPAT2001 “Life is too short for MPI.”
Hadoop͕ղܾͨ͠ͷ • Պֶܭࢉ͚Ͱͳ͘େنσʔλʹಛԽ • ڊେσʔλͷஔͱॲཧͷ࣮ߦΛࣗಈཧ • HDFSͰͷࣗಈࢄஔͱɺஔॴͰMAPॲཧ
HadoopҎ߱ͷ৽ͨͳχʔζ • Hadoop / Hiveεϧʔϓοτॏࢹͷόονܥ • σʔλαΠΤϯςΟετͷχʔζΠϯλϥΫςΟϒͳ ੳɾϦΞϧλΠϜॲཧ • ॲཧֻ͚ͯ࣌ؒͪݫ͍͠
• Hadoop, Hiveߴ৴པੑͷ֬อͱҾ͖͑ʹதؒσʔλ ͷDisk I/O͕ϘτϧωοΫʹ • αʔόͨΓͷϝϞϦ༰ྔ૿େ
HadoopޙͷϓϩμΫτ • HiveͷΦϯϝϞϦߴԽ • ϦΞϧλΠϜͷετϦʔ Ϝσʔλॲཧ • ෳͷσʔλιʔε / DB
ʹ·͕ͨͬͯͷߴूܭ • λεΫ࣮ߦΛ࠷దԽ͠ϨΠςϯγΛ࣮ݱ
Spark • ൚༻ͷࢄϓϩάϥϛϯάڥ • RDDΛجૅʹ͓͍ͨεέϧτϯฒྻϓϩάϥϛϯάڥ • ΦϯϝϞϦͷRDDΛ༻͍Δ͜ͱͰɺϨΠςϯγʔͷ ࢄܭࢉΛ࣮ݱ • ϝϞϦʹΒͳ͍ͷDiskʹอଘ
• RDDʹର͢Δૢ࡞ΛΈ߹ΘͤΔ͜ͱͰɺػցֶशε τϦʔϜσʔλॲཧΛ࣮ݱ
RDDʹର͢Δجຊԋࢉ • ScalaͷSeqॲཧͷߴ֊ؔ+α͕ࢄ࣮ߦ • map, flatMap, filter, sort, union, zip
• reduce, fold, reduceByKey, groupBy, groupByKey, count cogroup, cross • join, leftOuterJoin, rightOuterJoin • sample, take, first, partitionBy, mapWith, pipe, save • etc….
RDDͷσʔλϩʔΧϦςΟ • λεΫͷ࣮ߦॴɾॱংσʔλɾιʔεͷ ஔॴΛݩʹ࠷దͳDAGදݱͰཧ )%'4 3%% 3%% NBQ NBQ NBQ
NBQ 3%% 3FEVDF
RDDͷোੑ • RDDͷ֤ཁૉ͕ࣗͲͷΑ͏ͳܦ࿏Ͱੜ ͞Ε͔ͨه )%'4 NBQ NBQ ☓ഁଛ )%'4 NBQ
NBQ NBQ ࠶ඞཁʹͳͬͨ࣌ɺσʔλɾιʔε͔Β࠶ੜ
Sparkʹ͍ͭͯ(2) • RDDʹର͢Δࢄॲཧج൫ͷ্ʹҎԼΛ࣮ • σʔλετϦʔϜॲཧ(Spark Streaming) • ࢄSQL(SparkSQL) • ࢄػցֶशϥΠϒϥϦ(Mllib)
• ࢄάϥϑॲཧϥΠϒϥϦ(GraphX)
PySpark + IPython Notebook • PySparkIPython্Ͱ࣮ߦՄೳ • AWSͳΒɺίϚϯυϥΠϯ1ൃͰΫϥελߏஙՄೳ • Spark
on EMR(YARNରԠ)Λಈ͔͢ • http://qiita.com/shunsukeaihara/items/1524b66579e91d1cf7cf
• ఆظόονܥfluentd -> RedshiftͰॲཧ • ΞυϗοΫͳϩάੳFluentd -> S3 -> Spark
• S3্ͷେྔͷϑΝΠϧΛखܰʹॲཧՄೳ GunosyͷSparkϢʔεέʔε "1*αʔό 4QBSLPO"84&.3 3FETIJGU$MVTUFS
GunosyͷSparkϢʔεέʔε(1) • CloudTrailsͷϩά͔ΒΘΕ͍ͯΔCredentialΛ୳ͯ͠ ௵͢ͱ͔… • େྔͷJSONϑΝΠϧΛಡΈࠐΜͰHiveQLΛ࣮ߦ EBUBTDUFYU'JMF TCVDLFU@OBNFQBUI H[
IJWFQZTQBSLTRM)JWF$POUFYU TD IUIJWFKTPO3%% EBUB IUSFHJTUFS5FNQ5BCMF USBJMMT IUDBDIF5BCMF USBJMMT IJWFTRM 4&-&$5%*45*/$5SFDPSEVTFS*EFOUJUZBDDFTT,FZ*E '30.USBJMMT-"5&3"-7*&8FYQMPEF 3FDPSET TBTSFDPSE
GunosyͷSparkϢʔεέʔε(2) • Ϣʔβͷهࣄϩά͔Βͷੑผྨ • Ϣʔβຖʹclickͨ͠هࣄͷidΛListΛcsvͰS3ʹอଘ • TF-IDFͰॏΈ͚ͭ TD4QBSL$POUFYU NBMFTDUFYU'JMF
lTCVDLFUQBUINBMF@ H[l GFNBMFTDUFYU'JMF lTCVDLFUQBUINBMF@ H[l UG)BTIJOH5' OVN'FBUVSFT NBMFNBMFNBQ MBNCEBYUGUSBOTGPSN YTQMJU l z GFNBMFNBMFNBQ MBNCEBYUGUSBOTGPSN YTQMJU l z JEG*%' JEG@NPEFMJEGpU NBMFVOJPO GFNBMF NBMFJEG@NPEFMUSBOTGPSN NBMF GFNBMFJEG@NPEFMUSBOTGPSN GFNBMF
GunosyͷSparkϢʔεέʔε(2) • Ϣʔβͷهࣄϩά͔Βͷੑผྨ • LabeledPointʹม͠ϩδεςΟοΫճؼͰֶश/ ྨ NBMFNBMFNBQ MBNCEBY-BCFMFE1PJOU Y
GFNBMFGFNBMFNBQ MBNCEBY-BCFMFE1PJOU Y USBJOJOHNBMFVOJPO GFNBMF USBJOJOHDBDIF NPEFM-PHJTUJD3FHSFTTJPO8JUI4(%USBJO USBJOJOH
GunosyͷSparkϢʔεέʔε(2) • Ϣʔβͷهࣄϩά͔Βͷੑผྨ • ઌ಄͕ϢʔβID, ͦΕҎ͕߱هࣄIDͷϦετ͔Βਪఆ EFGQBSTF Y EBUB<JOU
J GPSJJOYTQMJU l z > SFUVSO-BCFMFE1PJOU EBUB<> EBUB<> VOLOPXOTDUFYU'JMF lTCVDLFUQBUIVOLOPXO@ H[l VOLOPXOVOLOPXONBQ MBNCEBYUGUSBOTGPSN YTQMJU l z VOLOPXOVOLOPXONBQ MBNCEBY Y<> JEG@NPEFMUPSBOTGPSN UGUSBOTGPSNY<> VOLOPXONBQ MBNCEBY Y<> NPEFMQSFEJDU Y<> DPMMFDU
Pyspark͓ख͚ܰͩͲ… • PythonͷؔΛPickleͯ͠ࢄ࣮ߦ͢ΔͷͰ͍Ζ͍Ζ͍ • JavaͷϥΠϒϥϦ(kuromoji)Λར༻͍ͨ͠߹Scala ͷϥούʔ + py4jͷϥούʔ͕ඞཁ • Scala͔ΒͳΒͦͷ··͑Δ
• ؤுͬͯΈ͚ͨͲ࠳ંɻpy4jͱʹ͔ͭ͘Β͍ • Spark༻్ఔͳΒScalaͷֶशίετ͍ • ͱ͍͑sbt໘͚ͩͲ…
Pythonͷࢄॲཧڥ
PythonͷࢄॲཧϥΠϒϥϦ • Ϋϥελܭࢉ༻ • PyRC, dispy, Pyro4(GensimͷLSI, LDAͷࢄԽόοΫΤϯυʹར༻) • ࢄλεΫΩϡʔ
• Celery : σίϨʔλΛ͚ͭΔ͚ͩͰؔ୯ҐͰඇಉظࢄԽ • IPython Cluster: ؆୯ͳλεΫࢄ༻ • Spartan: Numpy arrayͷZeroMQʹΑΔࢄԽ(SparkͷRDDΠϯεύΠΞ) • Disco: PythonMapReduceϑϨʔϜϫʔΫ
GunosyͷPythonࢄॲཧڥ • ػցֶशͷαʔϏε࿈ܞλεΫฒྻ(ฒྻετϦʔϜॲཧ)͕ॏ ཁͰφΠʔϒͳࢄॲཧͰ͍͍ͨͯͳ͍(ex. Jubatus) • aws্ͩͱجຊσʔλશͯS3ʹूੵ • λεΫཧͱϦτϥΠCelery(AMQP)ʹͤΔ •
ϫʔΧʔͷσϓϩΠChef + OpsworksͰશࣗಈԽ • ΦϯϥΠϯֶशͷࢄԽparameter iterative mixing • EMΞϧΰϦζϜͷࢄԽσʔλΛਫฏࢄͯ͠ಠཱʹܭࢉͨ͠ ύϥϝʔλͷฏۉΛऔΔ
• هࣄऩूϢʔβຖͷਪનΛϫʔΧʔʹόϥϚΩ GunosyͷPythonࢄॲཧڥ هࣄΫϩʔϥʔ DFMFSZXPSLFS ਪનΤϯδϯ DFMFSZXPSLFS هࣄΫϦοΫϩά ίϯτϩʔϥ EKBOHPDFMFSZ
·ͱΊ • Sparkͷ؊RDDͱ͍͏σʔλߏͱεέϧτϯฒྻϕʔ εͷ൚༻తͳฒྻϓϩάϥϛϯάڥ • Python͔Βͷखܰʹࢄॲཧͱࢄػցֶश͕͑ͯศར • ͰPython͔Βෳࡶͳ͜ͱΛ͠Α͏ͱ͢ΔͱຊʹΩπΠ ͷͰScalaͰॻ͖·͠ΐ͏ •
Ͳ͏ͯ͠Python͕ྑ͍ͳΒଞͷPythonͷࢄॲཧΤ ίγεςϜΛݕ౼͠·͠ΐ͏