Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
詳解Spark出版イベントでの発表資料
Search
Norihiro Shimoda
May 11, 2016
3
2.6k
詳解Spark出版イベントでの発表資料
http://connpass.com/event/30375/
Norihiro Shimoda
May 11, 2016
Tweet
Share
More Decks by Norihiro Shimoda
See All by Norihiro Shimoda
ぼくのかんがえたさいきょうの機械学習プロジェクト進行法(PoC/デモ編)
rindai87
10
19k
FindYourCandyでの転移学習の話
rindai87
4
15k
Google I/O 報告会(ML)担当
rindai87
3
9.4k
最近のTensorFlowの話
rindai87
1
1.4k
TensorFlowとGCPの簡単な紹介
rindai87
3
4.4k
TensorFlowの基礎
rindai87
2
4.3k
TFv1.0の概要+HighLevelAPI/Keras
rindai87
4
2.1k
How should engineers survive during AI era
rindai87
5
27k
Let's stand up against "Do Artificial Intelligence" with proper knowledge
rindai87
6
1.8k
Featured
See All Featured
Happy Clients
brianwarren
98
6.7k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
229
52k
Ruby is Unlike a Banana
tanoku
97
11k
The Pragmatic Product Professional
lauravandoore
31
6.3k
Site-Speed That Sticks
csswizardry
0
23
Stop Working from a Prison Cell
hatefulcrawdad
267
20k
Designing on Purpose - Digital PM Summit 2013
jponch
115
7k
Embracing the Ebb and Flow
colly
84
4.5k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
27
4.3k
Why You Should Never Use an ORM
jnunemaker
PRO
54
9.1k
A better future with KSS
kneath
238
17k
Navigating Team Friction
lara
183
14k
Transcript
σʔλੳۀքͷཧͱݱ࣮ ʙ4QBSLීٴͷྺ࢙Λఴ͑ͯʙ Լాྙେʢ!SJOEBJʣ
ࣗݾհ w גࣜձࣾϒϨΠϯύουͰٕज़ྖҬͷࡶ༻Λ୲ ͍ͯ͠·͢ w σʔλαΠΤϯςΟετͱΤϯδχΞ͕Ұମʹ ͳͬͨ৫ͮ͘Γʹᬏਐத w ࠷ۙϓϦηʔϧε࠾༻׆ಈͳΜ͔͍ͯ͠Δ ͷͰԿͰײ͕ɾɾɾ
ʮσʔλαΠΤϯεʯΈ͍ͨͳΩʔϫʔυͰ ॻ੶ϜοΫຊʹؔΘΒͤͯ͘ػձ͕͋Γ·ͨ͠ ݸਓతʹʮੳʯʮΤϯδχΞϦϯάʯʮϏδωεʯ ͷόϥϯεͷྑ͍ਓͰ͋Γ͍ͨͳɺͱࢥ͏ࠓ͜ͷ͝Ζ
ࣥචࢀՃͷܦҢ w લ৬ͰҰॹͩͬͨੴ͞Μʹ༠ΘΕͯ w ͪΐ͏Ͳࣗ4QBSLʹ࢝͠Ί͍ͯΔࠒʹ ੴ͞Μ4QBSLʹຊ֨తʹऔΓΜͰ͍Δͱ͍ ͏ۮવ͕ʂ w 4USBUB )BEPPQ8PSMEΛҰॹʹ८Γ·
ͨ͠
.BQ3ͷ5FE%VOOJOHͱͷҰຕ!TUSBUB
ݴ͍͍ͨ͜ͱ w 4QBSLొ࣌ʹࢲເͱϩϚϯΛݟ·ͨ͠ʂ w ͔͠͠ɺݱ࣮େมݫ͍͠ͷͰ͢ʂ w Έͳ͞Μɺͬͱ4QBSLؚΊͯେنσʔλ ػցֶशΛΨϯΨϯ͓͏ͱͯ͠େ͍ʹࠔͬͯ ͍ͩ͘͞ʂʂ
ڞஶऀͷํʑͷྨ w ࣄۀձࣾ w ϦΫϧʔτɺ%..DPNɺ4NBSU/FXT w ϕϯμʔʢϓϥοτϑΥʔϚʔʣ w )PSUPO8PSLT *#.
w ϕϯμʔʢίϯαϧɺ4*ܥʣ w ϒϨΠϯύου
ͦΕͰ4QBSLීٴͷ ྺ࢙ͱզʑͷาΈΛ ͓͠·͢
w 4QBSL͕".1-BCͷϓϩδΣΫτͱͯ͠ొ w ຊͩͱ͔֬·ͩ)BEPPQ͕ྲྀߦͩ͢͠લ͘ Β͍Ͱ͠ΐ͏͔ʁ
ʙ w ຊͰ8FCܥاۀΛத৺ʹ)BEPPQ͕ͲΜͲ ΜྲྀߦΓ͡Ί·ͨ͠ w ͳ͓ɺࣾͰଞͷ8FCܥاۀΑΓগ͠Εͯ )BEPPQͷಋೖ͕ࠒʹ࢝·ͬͨهԱ͕͋ Γ·͢
ࣾͷσʔλੳڥ w )BEPPQ)JWFΛத৺ʹਾ͑ͨσʔλੳج൫͕ ࡞ΒΕ͍ͯ·ͨ͠ w ͔͚࣌ؒͯॲཧ͍ͯͨ͠نͷσʔλΛΠ ϯλϥΫςΟϒʹॲཧͰ͖Δ͚ͩͰେײಈʂ
)JWFͱ.BIPVUΛར༻ͨ͠ ࠂ։ൃΛߦ͍·ͨ͠ ʮσʔλαΠΤϯςΟετཆಡຊʯ ʹͦͷΛࣥච͠·ͨ͠ ͋ɺ͔֬·ͩௐͷΑ͏Ͱ ୈ൛͕ग़Δͱ͔Ͱͳ͍ͱ͔ )BEPPQΛ׆༻ͨ͠ϓϩμΫτ ։ൃ
w 4QBSL͕"QBDIF4PGUXBSF'PVOEBUJPO "4' ʹدଃ͞Εɺ"QBDIF4QBSLͱͳΔ w %BUBCSJDLT͕ελʔτ w ຊͰ4QBSL·ͩ·ͩແ໊ͩͬͨͣ
w ͕ɺͲͪΒ͔ͱݴ͏ͱʮूܭʯͳײ͡ w େنσʔλʹର͢Δػցֶशͦͷଞͷχʔζ͕ ͦ͜·Ͱͳ͍ʁ w 8FCܥاۀͷϩάղੳͱ͔ϚʔέςΟϯά σʔλͷੳͱ͔ͦͷล͘Β͍ʁ w ͋·Γࣾ֎ʹཔΜͰΔέʔεଟ͘ͳͦ͞͏
େنσʔλॲཧͱݴ͑ )BEPPQͱ͍͏࣌ʹಥೖ
w ࢲͱ4QBSL͕ຊ֨తʹग़ձͬͨ w ৭ʑ͋ͬͯ4QBSLͰϏδωεΛ͍ͬͯΔΠϯ υاۀͷࢹؙग़ுʹ͍͖·ͨ͠ w ׂͱૣΊʹඈͼͭ͘Ϣʔβʔ͕ग़࢝Ί͖ͯͨ͘ Β͍ʁ
4QBSLͷୈҰҹ w 1ZUIPO3ͷΠϯλʔϑΣʔε͕͋Δʂ͢͝ ͍ʂ w ϦΞϧλΠϜॲཧಉ͡Α͏ʹॻ͍࣮ͯݱͰ͖Δ ͩͬͯʂ͍͢͝ʂ w ΤίγεςϜ͕Α͘ߟ͑ΒΕ͍ͯΔʂ͍͢͝ʂ
IUUQTPHJSBSEPUXPSEQSFTTDPNSEETBSFUIFOFXCZUFDPEFPGBQBDIFTQBSL
σʔλαΠΤϯςΟετΤ ϯδχΞ๊͍͑ͯΔฐࣾͷ Α͏ͳاۀऔΓΉ͖ʂ
ʹ͖߹ͬͨݱ࣮ w ϒϨΠϯύουࣾͱͯ͠ɼϏδωεϞσϧతʹ 4QBSLΛ࠾༻͍ͯ͘͠ϝϦοτ͕େ͖͘ͳ͍ w σʔλੳΛ֘اۀͷ֎෦Ͱҕୗ͞Εͯߦ ͏ձࣾͷ໋॓ͷ༷ͳɾɾɾ w ͦͷҰํͰɺࣄۀձࣾͷํʑ͕ͲΜͲΜࣄྫ Λग़ͯ͘͠Δɾɾɾ͏Β·͍͠ɾɾɾ
ʢ࠶ܝʣڞஶऀͷํʑͷྨ w ࣄۀձࣾ w ϦΫϧʔτɺ%..DPNɺ4NBSU/FXT w ϕϯμʔʢϓϥοτϑΥʔϚʔʣ w )PSUPO8PSLT *#.
w ϕϯμʔʢίϯαϧɺ4*ܥʣ w ϒϨΠϯύου
Կ͔৽͍ٕ͠ज़ ࣄۀձࣾ ϓϥοτϑΥʔϚʔ ίϯαϧ4* ϏδωεͱηοτͰߟ͑Δͱ ͨͿΜ͜Μͳײ͡Ͱ͢ ৽͍ٕ͠ज़͕ඞཁͱ ͞ΕΔ·ͰҰ൪ԕ͍
ϒϨΠϯύουͰ໘࣮ͨ͠ w ͦͦΫϥΠΞϯτʹ4QBSLͷΑ͏ͳϓϩμΫτΛ࠾ ༻͢Δχʔζ͕ඞཁ w 4QBSLͷಋೖ͔Βʢσʔλੳج൫ߏஙʣੳ·Ͱɺ ͱ͍͏χʔζΛ͍࣋ͬͯΔͷ΄ΜͷҰѲΓ w 4QBSLͷϝϦοτ͕ग़ͯ͘ΔͷશମͷΦϖϨʔγϣϯ ͱηοτͰ͋ΓɺΓग़͞ΕΔੳ෦ͷΈͰ30*త
ͳԸܙ͕গͳͯ͘ఏҊ͢ΔϝϦοτͱͱͯ͠ɺࣗࣾ ͷੳͷج൫ͱͯ͠ϝϦοτͱͱͯ͠ऑ͍ɾɾɾPS[
ͦ͜ͰࢲୡͲ͏͔ͨ͠ʁ w %FMUB$VCFͱ͍͏ࣗࣾ։ൃͷϓϩμΫτͰ࠾༻ ͠·ͨ͠ w ࣗࣾϓϩμΫτ͍࣋ͬͯΔͷ͕ٹ͍ͩͬͨ w ͜ͷลͷ͜ͷޙɺฐࣾࢣԬͷύʔτͰ ͕Ͱ͖Εͱ
༨ஊɿਂֶशͷొ w ਂֶशͷ౸དྷʹΑͬͯ·ͨগ͠ੳϏδωε ք۾ͷτϨϯυ͕มΘΓͭͭ͋Γ·͢ w ʮૣ͘ॲཧ͕Ͱ͖ͨํ͕ྑ͍ʯ͔Βʮ͕͔࣌ؒ ͔ͬͯྑ͍ʢͱ͍͏͔͔͔Δͷํͳ͍ʣ ͷͰ͔ͬ͠Γͱͨ͠Ϟσϧ͕࡞ΒΕΔ͜ͱ͕ॏ ཁʯ
ਓͰ͋ΔఔͰ͖͍ͯΔͱ͜ΖΛػցతʹΔ͜ͱͰ ޮԽΛਤΔ 30*ޮՌΛదਖ਼ʹଌΕΔΑ͏ʹࢿΛ͍ͨ͠ ੳػցֶशΛத৺ʹ͜Ε·Ͱશ͘ߟ͔͑ͭͳ͔ͬͨ ΠϊϕʔγϣϯΛى͍ͨ͜͠ʢىͤ͜ΔͰ͠ΐʁʣ ΒͷॴʹདྷΔ૬ஊ ͪΐͬͱ4QBSLͳͲΛఏҊ͍͢͠ڥʹͳ͖ͬͯͨʁ
·ͱΊ w ݱঢ়ɺϏδωεͱ͓ͯ͠ࣄͰ4QBSLΛΰϦΰ Ϧ৮Δػձ͕࣋ͪʹ͍͘ w ͜ΕΛଧഁ͢Δʹɺͬͱ4QBSLͷੳڥ Λඞཁͱ͠ಋೖΛݕ౼͍ͯͨ͠Γɺ࣮ࡍʹ࣋ͬ ͍ͯͯ׆༻ʹࠔ͍ͬͯΔΑ͏ͳاۀ͕ඞཁ w ͬͱ4QBSLΑྲྀߦͬͯ͘Εʂ
$IBOHFUP.PSPPLBTBO
σʔλϚωδϝϯτπʔϧ %FMUB$VCFͰͷ4QBSLར༻ ࢣԬҰ
ࣗݾհ w גࣜձࣾϒϨΠϯύουͰΤϯδχΞͯ͠·͢ w ݩʑ)BEPPQΓͨͯ͘ೖࣾ͠·͕ͨ͠ɺ͍ Ζ͍Ζͬͯ·͢ w ࠷ۙͳ͔ͥ0QFO*%$POOFDUͬͨೝূαʔ ό࡞ͬͯ·͢
4QBSLͱͷग़ձ͍ w ܰ͘৮ͬͨͷ͔Βɺຊ֨తʹۀͰ ͍ͩͨ͠ͷ͔Β w ͦͷੲʹ)BEPPQͷ/BUJWF"1*ͰͷΞϓϦέʔγϣ ϯΛ։ൃɾӡ༻ͯͨ͠ͷͰɺ1ZUIPO͑ͨΓ݁ ߹͕؆୯ʹग़དྷͨΓͰɺʮ͓͓ʔʯͬͯͳΔ
ຊ͢͜ͱ w %FMUB$VCFͱ͍͏ϓϩμΫτͰ4QBSLΛಋೖ͠ ͨΛ͠·͢
%FMUB$VCF w ϓϥΠϕʔτ%.13UPBTUFSͰอ͍࣋ͯ͠Δ ߦಈϩά͔ΒηάϝϯτΛ࡞͢Δπʔϧ
%FMUB$VCF w ։ൃ͔Β։࢝ɺਖ਼ࣜϦϦʔε w ։ൃ։͔࢝Β͘Β͍Ͱ͕͢ɺ͍Ζ͍ΖͳมભΛ ؚΊ͍ͯͯ͜͠͏͔ͱࢥ͍·͢
ॳظɿʙ w ߦಈϩάΛखಈͰݕࡧͯ͠ηάϝϯτΛ࡞ w ηάϝϯτಉ࢜Λֻ͚͋ΘͤͨΓͱ͔৭ʑ
ॳظɿʙ w "84্ʹ1SFTUPΫϥελΛΜͰੜϩάݕࡧ w σʔλαΠζతʹ5# Web Servers Elastic Load Balancing
Presto Cluster Coordinator Workers
தظɿʙ w ػೳతʹϚχΞοΫ࿏ઢ w ࠂӡ༻ͷख͕͔͔ؒΒͳ͍Α͏ͳϥΠτͳײ ͡ͷ࿏ઢ͕ཁͱͯ͠ଟʑ͋Γ
தظɿʙ w ͦ͜ͰҰ࿈ͷྲྀΕΛࣗಈԽͯ͠Έͨ w ͜ͷϢʔβͷΫϥελϦϯάʹ4QBSLΛ༻ ࣗಈͰϢʔβΛ ΫϥελϦϯά ࣗಈͰ"#ςετ ͯ͠࠷దԽ
தظɿʙ w &.3ͷ:"3/্Ͱ4QBSL w &5-ɾूܭɿ4QBSL$PSFɺ4QBSL42- w ΫϥελϦϯάɿ.-MJC Amazon EMR Web
Servers Elastic Load Balancing Presto Cluster Coordinator Workers
தظɿʙ w ͳͥ4QBSLΛબΜ͔ͩʁ w ͍ɺແཧཧͰ͢ w ͨͩɺಋೖ͠ऴΘͬͯΈΔͱɺ&5-ɾूܭɾػցֶ श͕ಉҰͷϓϥοτϑΥʔϜɾݴޠɾϑϨʔϜϫʔ ΫͰ࣮Ͱ͖ͨͷେ͖͔ͬͨͰ͢ w
։ൃظؒݕূؚΊͯϲ݄ఔ
ݱࡏʙ w :"3/ͱ1SFTUPͰΫϥελͭ͋ΔͷΠϠ w 1SFTUPແͯ͘͠4QBSL42-ʢ+%#$ʣʁ Amazon EMR Web Servers Elastic
Load Balancing Presto Cluster Coordinator Workers
ݱࡏʙ w 1SFTUPແ͍ͤͯ͘·ͤΜɻɻɻ w 4QBSLγϟοϑϧ࣌ʹσΟεΫॻࠐ͕͋ΔͷͰɺ ΦϯϥΠϯ͔ΒͷݕࡧੑೳͰ1SFTUPʹ܉
ݱࡏʙ w ࠷ऴతʹ&.3্ʹ1SFTUPΫϥελΛҠಈͤ͞ ͯӡ༻ͯ͠·͢ Web Servers Elastic Load Balancing Amazon
EMR ΦϯϥΠϯܥ όονܥ
·ͱΊ w %FMUB$VCFͱ͍͏ϓϩμΫτͰ4QBSLΛಋೖͯ͠Έ ͨΛ͠·ͨ͠ w 4QBSL৭ʑͳίϯϙʔωϯτ͕ඪ४Ͱೖ͍ͬͯͯɺ .BQ3FEVDFʹൺ૬ෑډ͕͘ͳ͍ͬͯ·͢ w ͬͱ4QBSLΑྲྀߦͬͯ͘Εʂ