Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
詳解Spark出版イベントでの発表資料
Search
Norihiro Shimoda
May 11, 2016
3
2.7k
詳解Spark出版イベントでの発表資料
http://connpass.com/event/30375/
Norihiro Shimoda
May 11, 2016
Tweet
Share
More Decks by Norihiro Shimoda
See All by Norihiro Shimoda
ぼくのかんがえたさいきょうの機械学習プロジェクト進行法(PoC/デモ編)
rindai87
10
19k
FindYourCandyでの転移学習の話
rindai87
4
15k
Google I/O 報告会(ML)担当
rindai87
3
9.5k
最近のTensorFlowの話
rindai87
1
1.5k
TensorFlowとGCPの簡単な紹介
rindai87
3
4.6k
TensorFlowの基礎
rindai87
2
4.4k
TFv1.0の概要+HighLevelAPI/Keras
rindai87
4
2.1k
How should engineers survive during AI era
rindai87
5
27k
Let's stand up against "Do Artificial Intelligence" with proper knowledge
rindai87
6
1.8k
Featured
See All Featured
Documentation Writing (for coders)
carmenintech
73
5k
Agile that works and the tools we love
rasmusluckow
329
21k
How to train your dragon (web standard)
notwaldorf
96
6.1k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.6k
Unsuck your backbone
ammeep
671
58k
Build The Right Thing And Hit Your Dates
maggiecrowley
37
2.8k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
34
3.1k
It's Worth the Effort
3n
185
28k
Optimizing for Happiness
mojombo
379
70k
Producing Creativity
orderedlist
PRO
346
40k
Transcript
σʔλੳۀքͷཧͱݱ࣮ ʙ4QBSLීٴͷྺ࢙Λఴ͑ͯʙ Լాྙେʢ!SJOEBJʣ
ࣗݾհ w גࣜձࣾϒϨΠϯύουͰٕज़ྖҬͷࡶ༻Λ୲ ͍ͯ͠·͢ w σʔλαΠΤϯςΟετͱΤϯδχΞ͕Ұମʹ ͳͬͨ৫ͮ͘Γʹᬏਐத w ࠷ۙϓϦηʔϧε࠾༻׆ಈͳΜ͔͍ͯ͠Δ ͷͰԿͰײ͕ɾɾɾ
ʮσʔλαΠΤϯεʯΈ͍ͨͳΩʔϫʔυͰ ॻ੶ϜοΫຊʹؔΘΒͤͯ͘ػձ͕͋Γ·ͨ͠ ݸਓతʹʮੳʯʮΤϯδχΞϦϯάʯʮϏδωεʯ ͷόϥϯεͷྑ͍ਓͰ͋Γ͍ͨͳɺͱࢥ͏ࠓ͜ͷ͝Ζ
ࣥචࢀՃͷܦҢ w લ৬ͰҰॹͩͬͨੴ͞Μʹ༠ΘΕͯ w ͪΐ͏Ͳࣗ4QBSLʹ࢝͠Ί͍ͯΔࠒʹ ੴ͞Μ4QBSLʹຊ֨తʹऔΓΜͰ͍Δͱ͍ ͏ۮવ͕ʂ w 4USBUB )BEPPQ8PSMEΛҰॹʹ८Γ·
ͨ͠
.BQ3ͷ5FE%VOOJOHͱͷҰຕ!TUSBUB
ݴ͍͍ͨ͜ͱ w 4QBSLొ࣌ʹࢲເͱϩϚϯΛݟ·ͨ͠ʂ w ͔͠͠ɺݱ࣮େมݫ͍͠ͷͰ͢ʂ w Έͳ͞Μɺͬͱ4QBSLؚΊͯେنσʔλ ػցֶशΛΨϯΨϯ͓͏ͱͯ͠େ͍ʹࠔͬͯ ͍ͩ͘͞ʂʂ
ڞஶऀͷํʑͷྨ w ࣄۀձࣾ w ϦΫϧʔτɺ%..DPNɺ4NBSU/FXT w ϕϯμʔʢϓϥοτϑΥʔϚʔʣ w )PSUPO8PSLT *#.
w ϕϯμʔʢίϯαϧɺ4*ܥʣ w ϒϨΠϯύου
ͦΕͰ4QBSLීٴͷ ྺ࢙ͱզʑͷาΈΛ ͓͠·͢
w 4QBSL͕".1-BCͷϓϩδΣΫτͱͯ͠ొ w ຊͩͱ͔֬·ͩ)BEPPQ͕ྲྀߦͩ͢͠લ͘ Β͍Ͱ͠ΐ͏͔ʁ
ʙ w ຊͰ8FCܥاۀΛத৺ʹ)BEPPQ͕ͲΜͲ ΜྲྀߦΓ͡Ί·ͨ͠ w ͳ͓ɺࣾͰଞͷ8FCܥاۀΑΓগ͠Εͯ )BEPPQͷಋೖ͕ࠒʹ࢝·ͬͨهԱ͕͋ Γ·͢
ࣾͷσʔλੳڥ w )BEPPQ)JWFΛத৺ʹਾ͑ͨσʔλੳج൫͕ ࡞ΒΕ͍ͯ·ͨ͠ w ͔͚࣌ؒͯॲཧ͍ͯͨ͠نͷσʔλΛΠ ϯλϥΫςΟϒʹॲཧͰ͖Δ͚ͩͰେײಈʂ
)JWFͱ.BIPVUΛར༻ͨ͠ ࠂ։ൃΛߦ͍·ͨ͠ ʮσʔλαΠΤϯςΟετཆಡຊʯ ʹͦͷΛࣥච͠·ͨ͠ ͋ɺ͔֬·ͩௐͷΑ͏Ͱ ୈ൛͕ग़Δͱ͔Ͱͳ͍ͱ͔ )BEPPQΛ׆༻ͨ͠ϓϩμΫτ ։ൃ
w 4QBSL͕"QBDIF4PGUXBSF'PVOEBUJPO "4' ʹدଃ͞Εɺ"QBDIF4QBSLͱͳΔ w %BUBCSJDLT͕ελʔτ w ຊͰ4QBSL·ͩ·ͩແ໊ͩͬͨͣ
w ͕ɺͲͪΒ͔ͱݴ͏ͱʮूܭʯͳײ͡ w େنσʔλʹର͢Δػցֶशͦͷଞͷχʔζ͕ ͦ͜·Ͱͳ͍ʁ w 8FCܥاۀͷϩάղੳͱ͔ϚʔέςΟϯά σʔλͷੳͱ͔ͦͷล͘Β͍ʁ w ͋·Γࣾ֎ʹཔΜͰΔέʔεଟ͘ͳͦ͞͏
େنσʔλॲཧͱݴ͑ )BEPPQͱ͍͏࣌ʹಥೖ
w ࢲͱ4QBSL͕ຊ֨తʹग़ձͬͨ w ৭ʑ͋ͬͯ4QBSLͰϏδωεΛ͍ͬͯΔΠϯ υاۀͷࢹؙग़ுʹ͍͖·ͨ͠ w ׂͱૣΊʹඈͼͭ͘Ϣʔβʔ͕ग़࢝Ί͖ͯͨ͘ Β͍ʁ
4QBSLͷୈҰҹ w 1ZUIPO3ͷΠϯλʔϑΣʔε͕͋Δʂ͢͝ ͍ʂ w ϦΞϧλΠϜॲཧಉ͡Α͏ʹॻ͍࣮ͯݱͰ͖Δ ͩͬͯʂ͍͢͝ʂ w ΤίγεςϜ͕Α͘ߟ͑ΒΕ͍ͯΔʂ͍͢͝ʂ
IUUQTPHJSBSEPUXPSEQSFTTDPNSEETBSFUIFOFXCZUFDPEFPGBQBDIFTQBSL
σʔλαΠΤϯςΟετΤ ϯδχΞ๊͍͑ͯΔฐࣾͷ Α͏ͳاۀऔΓΉ͖ʂ
ʹ͖߹ͬͨݱ࣮ w ϒϨΠϯύουࣾͱͯ͠ɼϏδωεϞσϧతʹ 4QBSLΛ࠾༻͍ͯ͘͠ϝϦοτ͕େ͖͘ͳ͍ w σʔλੳΛ֘اۀͷ֎෦Ͱҕୗ͞Εͯߦ ͏ձࣾͷ໋॓ͷ༷ͳɾɾɾ w ͦͷҰํͰɺࣄۀձࣾͷํʑ͕ͲΜͲΜࣄྫ Λग़ͯ͘͠Δɾɾɾ͏Β·͍͠ɾɾɾ
ʢ࠶ܝʣڞஶऀͷํʑͷྨ w ࣄۀձࣾ w ϦΫϧʔτɺ%..DPNɺ4NBSU/FXT w ϕϯμʔʢϓϥοτϑΥʔϚʔʣ w )PSUPO8PSLT *#.
w ϕϯμʔʢίϯαϧɺ4*ܥʣ w ϒϨΠϯύου
Կ͔৽͍ٕ͠ज़ ࣄۀձࣾ ϓϥοτϑΥʔϚʔ ίϯαϧ4* ϏδωεͱηοτͰߟ͑Δͱ ͨͿΜ͜Μͳײ͡Ͱ͢ ৽͍ٕ͠ज़͕ඞཁͱ ͞ΕΔ·ͰҰ൪ԕ͍
ϒϨΠϯύουͰ໘࣮ͨ͠ w ͦͦΫϥΠΞϯτʹ4QBSLͷΑ͏ͳϓϩμΫτΛ࠾ ༻͢Δχʔζ͕ඞཁ w 4QBSLͷಋೖ͔Βʢσʔλੳج൫ߏஙʣੳ·Ͱɺ ͱ͍͏χʔζΛ͍࣋ͬͯΔͷ΄ΜͷҰѲΓ w 4QBSLͷϝϦοτ͕ग़ͯ͘ΔͷશମͷΦϖϨʔγϣϯ ͱηοτͰ͋ΓɺΓग़͞ΕΔੳ෦ͷΈͰ30*త
ͳԸܙ͕গͳͯ͘ఏҊ͢ΔϝϦοτͱͱͯ͠ɺࣗࣾ ͷੳͷج൫ͱͯ͠ϝϦοτͱͱͯ͠ऑ͍ɾɾɾPS[
ͦ͜ͰࢲୡͲ͏͔ͨ͠ʁ w %FMUB$VCFͱ͍͏ࣗࣾ։ൃͷϓϩμΫτͰ࠾༻ ͠·ͨ͠ w ࣗࣾϓϩμΫτ͍࣋ͬͯΔͷ͕ٹ͍ͩͬͨ w ͜ͷลͷ͜ͷޙɺฐࣾࢣԬͷύʔτͰ ͕Ͱ͖Εͱ
༨ஊɿਂֶशͷొ w ਂֶशͷ౸དྷʹΑͬͯ·ͨগ͠ੳϏδωε ք۾ͷτϨϯυ͕มΘΓͭͭ͋Γ·͢ w ʮૣ͘ॲཧ͕Ͱ͖ͨํ͕ྑ͍ʯ͔Βʮ͕͔࣌ؒ ͔ͬͯྑ͍ʢͱ͍͏͔͔͔Δͷํͳ͍ʣ ͷͰ͔ͬ͠Γͱͨ͠Ϟσϧ͕࡞ΒΕΔ͜ͱ͕ॏ ཁʯ
ਓͰ͋ΔఔͰ͖͍ͯΔͱ͜ΖΛػցతʹΔ͜ͱͰ ޮԽΛਤΔ 30*ޮՌΛదਖ਼ʹଌΕΔΑ͏ʹࢿΛ͍ͨ͠ ੳػցֶशΛத৺ʹ͜Ε·Ͱશ͘ߟ͔͑ͭͳ͔ͬͨ ΠϊϕʔγϣϯΛى͍ͨ͜͠ʢىͤ͜ΔͰ͠ΐʁʣ ΒͷॴʹདྷΔ૬ஊ ͪΐͬͱ4QBSLͳͲΛఏҊ͍͢͠ڥʹͳ͖ͬͯͨʁ
·ͱΊ w ݱঢ়ɺϏδωεͱ͓ͯ͠ࣄͰ4QBSLΛΰϦΰ Ϧ৮Δػձ͕࣋ͪʹ͍͘ w ͜ΕΛଧഁ͢Δʹɺͬͱ4QBSLͷੳڥ Λඞཁͱ͠ಋೖΛݕ౼͍ͯͨ͠Γɺ࣮ࡍʹ࣋ͬ ͍ͯͯ׆༻ʹࠔ͍ͬͯΔΑ͏ͳاۀ͕ඞཁ w ͬͱ4QBSLΑྲྀߦͬͯ͘Εʂ
$IBOHFUP.PSPPLBTBO
σʔλϚωδϝϯτπʔϧ %FMUB$VCFͰͷ4QBSLར༻ ࢣԬҰ
ࣗݾհ w גࣜձࣾϒϨΠϯύουͰΤϯδχΞͯ͠·͢ w ݩʑ)BEPPQΓͨͯ͘ೖࣾ͠·͕ͨ͠ɺ͍ Ζ͍Ζͬͯ·͢ w ࠷ۙͳ͔ͥ0QFO*%$POOFDUͬͨೝূαʔ ό࡞ͬͯ·͢
4QBSLͱͷग़ձ͍ w ܰ͘৮ͬͨͷ͔Βɺຊ֨తʹۀͰ ͍ͩͨ͠ͷ͔Β w ͦͷੲʹ)BEPPQͷ/BUJWF"1*ͰͷΞϓϦέʔγϣ ϯΛ։ൃɾӡ༻ͯͨ͠ͷͰɺ1ZUIPO͑ͨΓ݁ ߹͕؆୯ʹग़དྷͨΓͰɺʮ͓͓ʔʯͬͯͳΔ
ຊ͢͜ͱ w %FMUB$VCFͱ͍͏ϓϩμΫτͰ4QBSLΛಋೖ͠ ͨΛ͠·͢
%FMUB$VCF w ϓϥΠϕʔτ%.13UPBTUFSͰอ͍࣋ͯ͠Δ ߦಈϩά͔ΒηάϝϯτΛ࡞͢Δπʔϧ
%FMUB$VCF w ։ൃ͔Β։࢝ɺਖ਼ࣜϦϦʔε w ։ൃ։͔࢝Β͘Β͍Ͱ͕͢ɺ͍Ζ͍ΖͳมભΛ ؚΊ͍ͯͯ͜͠͏͔ͱࢥ͍·͢
ॳظɿʙ w ߦಈϩάΛखಈͰݕࡧͯ͠ηάϝϯτΛ࡞ w ηάϝϯτಉ࢜Λֻ͚͋ΘͤͨΓͱ͔৭ʑ
ॳظɿʙ w "84্ʹ1SFTUPΫϥελΛΜͰੜϩάݕࡧ w σʔλαΠζతʹ5# Web Servers Elastic Load Balancing
Presto Cluster Coordinator Workers
தظɿʙ w ػೳతʹϚχΞοΫ࿏ઢ w ࠂӡ༻ͷख͕͔͔ؒΒͳ͍Α͏ͳϥΠτͳײ ͡ͷ࿏ઢ͕ཁͱͯ͠ଟʑ͋Γ
தظɿʙ w ͦ͜ͰҰ࿈ͷྲྀΕΛࣗಈԽͯ͠Έͨ w ͜ͷϢʔβͷΫϥελϦϯάʹ4QBSLΛ༻ ࣗಈͰϢʔβΛ ΫϥελϦϯά ࣗಈͰ"#ςετ ͯ͠࠷దԽ
தظɿʙ w &.3ͷ:"3/্Ͱ4QBSL w &5-ɾूܭɿ4QBSL$PSFɺ4QBSL42- w ΫϥελϦϯάɿ.-MJC Amazon EMR Web
Servers Elastic Load Balancing Presto Cluster Coordinator Workers
தظɿʙ w ͳͥ4QBSLΛબΜ͔ͩʁ w ͍ɺແཧཧͰ͢ w ͨͩɺಋೖ͠ऴΘͬͯΈΔͱɺ&5-ɾूܭɾػցֶ श͕ಉҰͷϓϥοτϑΥʔϜɾݴޠɾϑϨʔϜϫʔ ΫͰ࣮Ͱ͖ͨͷେ͖͔ͬͨͰ͢ w
։ൃظؒݕূؚΊͯϲ݄ఔ
ݱࡏʙ w :"3/ͱ1SFTUPͰΫϥελͭ͋ΔͷΠϠ w 1SFTUPແͯ͘͠4QBSL42-ʢ+%#$ʣʁ Amazon EMR Web Servers Elastic
Load Balancing Presto Cluster Coordinator Workers
ݱࡏʙ w 1SFTUPແ͍ͤͯ͘·ͤΜɻɻɻ w 4QBSLγϟοϑϧ࣌ʹσΟεΫॻࠐ͕͋ΔͷͰɺ ΦϯϥΠϯ͔ΒͷݕࡧੑೳͰ1SFTUPʹ܉
ݱࡏʙ w ࠷ऴతʹ&.3্ʹ1SFTUPΫϥελΛҠಈͤ͞ ͯӡ༻ͯ͠·͢ Web Servers Elastic Load Balancing Amazon
EMR ΦϯϥΠϯܥ όονܥ
·ͱΊ w %FMUB$VCFͱ͍͏ϓϩμΫτͰ4QBSLΛಋೖͯ͠Έ ͨΛ͠·ͨ͠ w 4QBSL৭ʑͳίϯϙʔωϯτ͕ඪ४Ͱೖ͍ͬͯͯɺ .BQ3FEVDFʹൺ૬ෑډ͕͘ͳ͍ͬͯ·͢ w ͬͱ4QBSLΑྲྀߦͬͯ͘Εʂ