Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Sparkによる分散処理 / 2015-01-16 PyData.Tokyo#3
shunsukeaihara
January 17, 2015
Technology
11
2.9k
Sparkによる分散処理 / 2015-01-16 PyData.Tokyo#3
shunsukeaihara
January 17, 2015
Tweet
Share
More Decks by shunsukeaihara
See All by shunsukeaihara
BONXを支える技術:発話区間検出(VAD)の話/Akerun & BONX Tech Talk
shunsukeaihara
4
6.4k
Goのnet.TCPConnの話/shibuya.go01
shunsukeaihara
2
610
Norikra in Gunosy Network Ads@Norikra meetup #2
shunsukeaihara
1
5.4k
LevelDB on S3 As A KVS
shunsukeaihara
1
2.3k
色恒常性仮説に基づく色補正ライブラリcolorcorrect / 2015-01-31-kantocv27
shunsukeaihara
3
1.7k
ゼロから始めた Gunosyアドサーバ開発運用記 / 2014-12-16-dots
shunsukeaihara
6
1k
Gunosy.Go#5 index/io/log
shunsukeaihara
0
84
Gunosy.go#2 package/compress
shunsukeaihara
0
66
Other Decks in Technology
See All in Technology
フロントエンド初心者が Blazorを使ってみた / 20220428 C#Tokyo
takahiro901
0
200
20220510_簡単にできるコスト異常検出(Cost Anomaly Detection) /jaws-ug-asa-cost-anomaly-detection-20220510
emiki
2
310
1年間のポストモーテム運用とそこから生まれたツール sre-advisor / SRE NEXT 2022
fujiwara3
5
2.6k
株式会社オプティム_採用会社紹介資料 / optim-recruit
optim
0
5.2k
TypeScript 4.7と型レベルプログラミング
uhyo
5
2.7k
Embedded SRE at Mercari
tcnksm
0
750
僕の Microsoft Teams (+α) 便利技紹介 2022年春
taichinakamura
0
2.2k
スタートアップ入社4日目までに考えたAWSのセキュリティ向上/ Startup AWS Security
shonansurvivors
3
2.4k
OSINT/GEOINT ワークショップ 20220514 古橋資料
furuhashilab
2
230
Web Intelligence and Visual Media Analytics
weblyzard
PRO
1
2.8k
~スタートアップの人たちに捧ぐ~ 監視再入門 in AWS
track3jyo
PRO
30
8.3k
Babylon.js v5 新機能の紹介
limes2018
0
350
Featured
See All Featured
VelocityConf: Rendering Performance Case Studies
addyosmani
316
22k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
103
16k
Designing on Purpose - Digital PM Summit 2013
jponch
106
5.6k
5 minutes of I Can Smell Your CMS
philhawksworth
196
18k
How to name files
jennybc
39
58k
How STYLIGHT went responsive
nonsquared
85
3.9k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
37
3.2k
Designing with Data
zakiwarfel
91
3.8k
GitHub's CSS Performance
jonrohan
1020
410k
Principles of Awesome APIs and How to Build Them.
keavy
113
15k
How to train your dragon (web standard)
notwaldorf
57
3.8k
Bash Introduction
62gerente
596
210k
Transcript
SparkʹΑΔࢄॲཧ (ͱPythonͰͷࢄॲཧ) Gunosy Inc. Shunsuke Aihara
ࣗݾհ • ҄൧ݪढ़հ (http://argmax.jp) @shunsukeaihara • GunosyͷϚωʔδϟʔ • ࠂ৴γεςϜͷ։ൃશମͱR&DܥΛ୲ •
ઐ: ܭࢉݴޠֶ • PythonͱඇಉظࢄγεςϜΛΉ • ը૾ॲཧɾԻ৴߸ॲཧͰ͍Ζ͍ΖϥΠϒϥϦ࡞ͬͯΔ • https://bitbucket.org/aihara
Agenda • Spark֓ཁ • ࢄॲཧ(ͱSpark)ͷ • GunosyͰͷSparkͷϢʔεέʔε • PythonͰͷࢄॲཧΤίγεςϜ
Sparkʹ͍ͭͯ(1) • HadoopͷΤίγεςϜ(HDFS, MESOS, YARN)ͱ࿈ܞ͢ΔΦϯϝϞ Ϧࢄॲཧܥ • Resillient Distributed Datasetsͱ͍͏োੑΛ࣋ͬͨࢄσʔλߏ
ʹର͢Δࢄϓϩάϥϛϯάڥ • RDDʹద༻͢ΔฒྻܭࢉΛɺߴ֊ؔͷνΣΠϯͷܗͰScalaɺ PythonͰ࣮ߦ • immutableͳσʔλߏ • RDDͷཁૉΫϥελͷΦϯϝϞϦʹࢄɾϨϓϦέʔγϣϯ • ഁଛɾϩετͨ͠σʔλӬଓԽͨ͠ݩσʔλ͔Β෮ݩ
Sparkʹ͍ͭͯ(2) • RDDʹର͢Δࢄॲཧج൫ͷ্ʹҎԼΛ࣮ • σʔλετϦʔϜॲཧ(Spark Streaming) • ࢄSQL(SparkSQL) • ࢄػցֶशϥΠϒϥϦ(Mllib)
• ࢄάϥϑॲཧϥΠϒϥϦ(GraphX)
ࢄॲཧ(ͱSpark)ͷ
େنσʔλࢄॲཧͷ؊ • ΫϥελϚωʔδϝϯτ • σʔλͷࢄஔͷࣗಈԽ • σʔλଟॏԽ/ฒྻReadʹΑΔߴԽ • σʔλϩʔΧϦςΟΛอͬͨܭࢉ •
োੑ / ࠶ૹɾ࠶ܭࢉॲཧ
HadoopʹࢸΔ·Ͱ • ෳࡶͳฒྻॲཧϝοηʔδύογϯάͰಠࣗʹ࣮͢Δͱେม • εέϧτϯฒྻϓϩάϥϛϯά(Cole, 1989) • සग़͢ΔฒྻܭࢉύλʔϯͷΈ߹ΘͤͰɺ༷ʑͳฒྻॲཧΛߏతʹߏங ͢ΔؔϓϩάϥϛϯάͷΈͱෳͷ࣮ •
σʔλฒྻεέϧτϯ(map, fold/reduce, filter, zip…) • σʔλͷҟͳΔ෦ʹɼಉ࣌ʹಉ͡ૢ࡞Λߦ͏ܭࢉύλʔϯ • λεΫฒྻεέϧτϯ(pipe, farm…) • σʔλͷετϦʔϜʹରͯ͠ɼͦΕͧΕܭࢉΛద༻ͨ͠σʔλετϦʔ ϜΛฦ͢ύλʔϯ
εέϧτϯฒྻϓϩάϥϛϯά މৼߐ ؠ࡚ӳ࠸ εέϧτϯฒྻϓϩάϥϛϯάใॲཧ 7PM /P QQ
HadoopҎલͷࢄॲཧ • MPI άϦουγΣϧΛ༻͍࣮ͯ • σʔλͷஔࣗͰϚωʔδ • ڞ༗ϝϞϦ͔ڞ༗FSʹࣗͰஔ͕લఏ • ڊେσʔλͷஔͱͯ໘
• োੑಠ࣮ࣗͰอূ • ϝϞϦʹࡌΓΒͳ͍σʔλΛѻ͏ͷ͍͠
T-shirts message@WOMPAT2001 “Life is too short for MPI.”
Hadoop͕ղܾͨ͠ͷ • Պֶܭࢉ͚Ͱͳ͘େنσʔλʹಛԽ • ڊେσʔλͷஔͱॲཧͷ࣮ߦΛࣗಈཧ • HDFSͰͷࣗಈࢄஔͱɺஔॴͰMAPॲཧ
HadoopҎ߱ͷ৽ͨͳχʔζ • Hadoop / Hiveεϧʔϓοτॏࢹͷόονܥ • σʔλαΠΤϯςΟετͷχʔζΠϯλϥΫςΟϒͳ ੳɾϦΞϧλΠϜॲཧ • ॲཧֻ͚ͯ࣌ؒͪݫ͍͠
• Hadoop, Hiveߴ৴པੑͷ֬อͱҾ͖͑ʹதؒσʔλ ͷDisk I/O͕ϘτϧωοΫʹ • αʔόͨΓͷϝϞϦ༰ྔ૿େ
HadoopޙͷϓϩμΫτ • HiveͷΦϯϝϞϦߴԽ • ϦΞϧλΠϜͷετϦʔ Ϝσʔλॲཧ • ෳͷσʔλιʔε / DB
ʹ·͕ͨͬͯͷߴूܭ • λεΫ࣮ߦΛ࠷దԽ͠ϨΠςϯγΛ࣮ݱ
Spark • ൚༻ͷࢄϓϩάϥϛϯάڥ • RDDΛجૅʹ͓͍ͨεέϧτϯฒྻϓϩάϥϛϯάڥ • ΦϯϝϞϦͷRDDΛ༻͍Δ͜ͱͰɺϨΠςϯγʔͷ ࢄܭࢉΛ࣮ݱ • ϝϞϦʹΒͳ͍ͷDiskʹอଘ
• RDDʹର͢Δૢ࡞ΛΈ߹ΘͤΔ͜ͱͰɺػցֶशε τϦʔϜσʔλॲཧΛ࣮ݱ
RDDʹର͢Δجຊԋࢉ • ScalaͷSeqॲཧͷߴ֊ؔ+α͕ࢄ࣮ߦ • map, flatMap, filter, sort, union, zip
• reduce, fold, reduceByKey, groupBy, groupByKey, count cogroup, cross • join, leftOuterJoin, rightOuterJoin • sample, take, first, partitionBy, mapWith, pipe, save • etc….
RDDͷσʔλϩʔΧϦςΟ • λεΫͷ࣮ߦॴɾॱংσʔλɾιʔεͷ ஔॴΛݩʹ࠷దͳDAGදݱͰཧ )%'4 3%% 3%% NBQ NBQ NBQ
NBQ 3%% 3FEVDF
RDDͷোੑ • RDDͷ֤ཁૉ͕ࣗͲͷΑ͏ͳܦ࿏Ͱੜ ͞Ε͔ͨه )%'4 NBQ NBQ ☓ഁଛ )%'4 NBQ
NBQ NBQ ࠶ඞཁʹͳͬͨ࣌ɺσʔλɾιʔε͔Β࠶ੜ
Sparkʹ͍ͭͯ(2) • RDDʹର͢Δࢄॲཧج൫ͷ্ʹҎԼΛ࣮ • σʔλετϦʔϜॲཧ(Spark Streaming) • ࢄSQL(SparkSQL) • ࢄػցֶशϥΠϒϥϦ(Mllib)
• ࢄάϥϑॲཧϥΠϒϥϦ(GraphX)
PySpark + IPython Notebook • PySparkIPython্Ͱ࣮ߦՄೳ • AWSͳΒɺίϚϯυϥΠϯ1ൃͰΫϥελߏஙՄೳ • Spark
on EMR(YARNରԠ)Λಈ͔͢ • http://qiita.com/shunsukeaihara/items/1524b66579e91d1cf7cf
• ఆظόονܥfluentd -> RedshiftͰॲཧ • ΞυϗοΫͳϩάੳFluentd -> S3 -> Spark
• S3্ͷେྔͷϑΝΠϧΛखܰʹॲཧՄೳ GunosyͷSparkϢʔεέʔε "1*αʔό 4QBSLPO"84&.3 3FETIJGU$MVTUFS
GunosyͷSparkϢʔεέʔε(1) • CloudTrailsͷϩά͔ΒΘΕ͍ͯΔCredentialΛ୳ͯ͠ ௵͢ͱ͔… • େྔͷJSONϑΝΠϧΛಡΈࠐΜͰHiveQLΛ࣮ߦ EBUBTDUFYU'JMF TCVDLFU@OBNFQBUI H[
IJWFQZTQBSLTRM)JWF$POUFYU TD IUIJWFKTPO3%% EBUB IUSFHJTUFS5FNQ5BCMF USBJMMT IUDBDIF5BCMF USBJMMT IJWFTRM 4&-&$5%*45*/$5SFDPSEVTFS*EFOUJUZBDDFTT,FZ*E '30.USBJMMT-"5&3"-7*&8FYQMPEF 3FDPSET TBTSFDPSE
GunosyͷSparkϢʔεέʔε(2) • Ϣʔβͷهࣄϩά͔Βͷੑผྨ • Ϣʔβຖʹclickͨ͠هࣄͷidΛListΛcsvͰS3ʹอଘ • TF-IDFͰॏΈ͚ͭ TD4QBSL$POUFYU NBMFTDUFYU'JMF
lTCVDLFUQBUINBMF@ H[l GFNBMFTDUFYU'JMF lTCVDLFUQBUINBMF@ H[l UG)BTIJOH5' OVN'FBUVSFT NBMFNBMFNBQ MBNCEBYUGUSBOTGPSN YTQMJU l z GFNBMFNBMFNBQ MBNCEBYUGUSBOTGPSN YTQMJU l z JEG*%' JEG@NPEFMJEGpU NBMFVOJPO GFNBMF NBMFJEG@NPEFMUSBOTGPSN NBMF GFNBMFJEG@NPEFMUSBOTGPSN GFNBMF
GunosyͷSparkϢʔεέʔε(2) • Ϣʔβͷهࣄϩά͔Βͷੑผྨ • LabeledPointʹม͠ϩδεςΟοΫճؼͰֶश/ ྨ NBMFNBMFNBQ MBNCEBY-BCFMFE1PJOU Y
GFNBMFGFNBMFNBQ MBNCEBY-BCFMFE1PJOU Y USBJOJOHNBMFVOJPO GFNBMF USBJOJOHDBDIF NPEFM-PHJTUJD3FHSFTTJPO8JUI4(%USBJO USBJOJOH
GunosyͷSparkϢʔεέʔε(2) • Ϣʔβͷهࣄϩά͔Βͷੑผྨ • ઌ಄͕ϢʔβID, ͦΕҎ͕߱هࣄIDͷϦετ͔Βਪఆ EFGQBSTF Y EBUB<JOU
J GPSJJOYTQMJU l z > SFUVSO-BCFMFE1PJOU EBUB<> EBUB<> VOLOPXOTDUFYU'JMF lTCVDLFUQBUIVOLOPXO@ H[l VOLOPXOVOLOPXONBQ MBNCEBYUGUSBOTGPSN YTQMJU l z VOLOPXOVOLOPXONBQ MBNCEBY Y<> JEG@NPEFMUPSBOTGPSN UGUSBOTGPSNY<> VOLOPXONBQ MBNCEBY Y<> NPEFMQSFEJDU Y<> DPMMFDU
Pyspark͓ख͚ܰͩͲ… • PythonͷؔΛPickleͯ͠ࢄ࣮ߦ͢ΔͷͰ͍Ζ͍Ζ͍ • JavaͷϥΠϒϥϦ(kuromoji)Λར༻͍ͨ͠߹Scala ͷϥούʔ + py4jͷϥούʔ͕ඞཁ • Scala͔ΒͳΒͦͷ··͑Δ
• ؤுͬͯΈ͚ͨͲ࠳ંɻpy4jͱʹ͔ͭ͘Β͍ • Spark༻్ఔͳΒScalaͷֶशίετ͍ • ͱ͍͑sbt໘͚ͩͲ…
Pythonͷࢄॲཧڥ
PythonͷࢄॲཧϥΠϒϥϦ • Ϋϥελܭࢉ༻ • PyRC, dispy, Pyro4(GensimͷLSI, LDAͷࢄԽόοΫΤϯυʹར༻) • ࢄλεΫΩϡʔ
• Celery : σίϨʔλΛ͚ͭΔ͚ͩͰؔ୯ҐͰඇಉظࢄԽ • IPython Cluster: ؆୯ͳλεΫࢄ༻ • Spartan: Numpy arrayͷZeroMQʹΑΔࢄԽ(SparkͷRDDΠϯεύΠΞ) • Disco: PythonMapReduceϑϨʔϜϫʔΫ
GunosyͷPythonࢄॲཧڥ • ػցֶशͷαʔϏε࿈ܞλεΫฒྻ(ฒྻετϦʔϜॲཧ)͕ॏ ཁͰφΠʔϒͳࢄॲཧͰ͍͍ͨͯͳ͍(ex. Jubatus) • aws্ͩͱجຊσʔλશͯS3ʹूੵ • λεΫཧͱϦτϥΠCelery(AMQP)ʹͤΔ •
ϫʔΧʔͷσϓϩΠChef + OpsworksͰશࣗಈԽ • ΦϯϥΠϯֶशͷࢄԽparameter iterative mixing • EMΞϧΰϦζϜͷࢄԽσʔλΛਫฏࢄͯ͠ಠཱʹܭࢉͨ͠ ύϥϝʔλͷฏۉΛऔΔ
• هࣄऩूϢʔβຖͷਪનΛϫʔΧʔʹόϥϚΩ GunosyͷPythonࢄॲཧڥ هࣄΫϩʔϥʔ DFMFSZXPSLFS ਪનΤϯδϯ DFMFSZXPSLFS هࣄΫϦοΫϩά ίϯτϩʔϥ EKBOHPDFMFSZ
·ͱΊ • Sparkͷ؊RDDͱ͍͏σʔλߏͱεέϧτϯฒྻϕʔ εͷ൚༻తͳฒྻϓϩάϥϛϯάڥ • Python͔Βͷखܰʹࢄॲཧͱࢄػցֶश͕͑ͯศར • ͰPython͔Βෳࡶͳ͜ͱΛ͠Α͏ͱ͢ΔͱຊʹΩπΠ ͷͰScalaͰॻ͖·͠ΐ͏ •
Ͳ͏ͯ͠Python͕ྑ͍ͳΒଞͷPythonͷࢄॲཧΤ ίγεςϜΛݕ౼͠·͠ΐ͏