詳解Spark出版イベントでの発表資料

by Norihiro Shimoda

Slide 1

Slide 1 text

σʔλ෼ੳۀքͷཧ૝ͱݱ࣮ ʙ4QBSLීٴͷྺ࢙Λఴ͑ͯʙ Լాྙେʢ!SJOEBJʣ

Slide 2

Slide 2 text

ࣗݾ঺հ w גࣜձࣾϒϨΠϯύουͰٕज़ྖҬͷࡶ༻Λ୲ ౰͍ͯ͠·͢ w σʔλαΠΤϯςΟετͱΤϯδχΞ͕Ұମʹ ͳͬͨ૊৫ͮ͘Γʹᬏਐத w ࠷ۙϓϦηʔϧε΍࠾༻׆ಈͳΜ͔΋͍ͯ͠Δ ͷͰ΍΍ԿͰ΋԰ײ͕ɾɾɾ

Slide 3

Slide 3 text

ʮσʔλαΠΤϯεʯΈ͍ͨͳΩʔϫʔυͰ ॻ੶΍ϜοΫຊʹؔΘΒͤͯ௖͘ػձ͕͋Γ·ͨ͠ ݸਓతʹ͸ʮ෼ੳʯʮΤϯδχΞϦϯάʯʮϏδωεʯ ͷόϥϯεͷྑ͍ਓͰ͋Γ͍ͨͳɺͱࢥ͏ࠓ೔͜ͷ͝Ζ

Slide 4

Slide 4 text

ࣥචࢀՃͷܦҢ w લ৬ͰҰॹͩͬͨੴ઒͞Μʹ༠ΘΕͯ w ͪΐ͏Ͳࣗ෼΋4QBSLʹ஫໨࢝͠Ί͍ͯΔࠒʹ ੴ઒͞Μ΋4QBSLʹຊ֨తʹऔΓ૊ΜͰ͍Δͱ͍ ͏ۮવ͕ʂ w 4USBUB)BEPPQ8PSMEΛҰॹʹ८Γ· ͨ͠

Slide 5

Slide 5 text

.BQ3ͷ5FE%VOOJOHͱͷҰຕ!TUSBUB

Slide 6

Slide 6 text

ݴ͍͍ͨ͜ͱ w 4QBSLొ৔࣌ʹࢲ͸ເͱϩϚϯΛݟ·ͨ͠ʂ w ͔͠͠ɺݱ࣮͸େมݫ͍͠ͷͰ͢ʂ w Έͳ͞Μɺ΋ͬͱ4QBSLؚΊͯେن໛σʔλ΍ ػցֶशΛΨϯΨϯ࢖͓͏ͱͯ͠େ͍ʹࠔͬͯ ͍ͩ͘͞ʂʂ

Slide 7

Slide 7 text

ڞஶऀͷํʑͷ෼ྨ w ࣄۀձࣾ w ϦΫϧʔτɺ%..DPNɺ4NBSU/FXT w ϕϯμʔʢϓϥοτϑΥʔϚʔʣ w )PSUPO8PSLT *#. w ϕϯμʔʢίϯαϧɺ4*ܥʣ w ϒϨΠϯύου

Slide 8

Slide 8 text

ͦΕͰ͸4QBSLීٴͷ ྺ࢙ͱզʑͷาΈΛ ͓࿩͠·͢

Slide 9

Slide 9 text

೥ w 4QBSL͕".1-BCͷϓϩδΣΫτͱͯ͠ొ৔ w ೔ຊͩͱ͔֬·ͩ)BEPPQ͕ྲྀߦͩ͢͠௚લ͘ Β͍Ͱ͠ΐ͏͔ʁ

Slide 10

Slide 10 text

೥ʙ೥ w ೔ຊͰ΋8FCܥاۀΛத৺ʹ)BEPPQ͕ͲΜͲ ΜྲྀߦΓ͸͡Ί·ͨ͠ w ͳ͓ɺ๭ࣾͰ͸ଞͷ8FCܥاۀΑΓগ͠஗Εͯ )BEPPQͷಋೖ͕೥ࠒʹ࢝·ͬͨهԱ͕͋ Γ·͢

Slide 11

Slide 11 text

๭ࣾͷσʔλ෼ੳ؀ڥ w )BEPPQ)JWFΛத৺ʹਾ͑ͨσʔλ෼ੳج൫͕ ࡞ΒΕ͍ͯ·ͨ͠ w ௕͔͚࣌ؒͯॲཧ͍ͯͨ͠ن໛ͷσʔλΛΠ ϯλϥΫςΟϒʹॲཧͰ͖Δ͚ͩͰେײಈʂ

Slide 12

Slide 12 text

)JWFͱ.BIPVUΛར༻ͨ͠ ޿ࠂ঎඼։ൃΛߦ͍·ͨ͠ ʮσʔλαΠΤϯςΟετཆ੒ಡຊʯ ʹͦͷ࿩Λࣥච͠·ͨ͠ ͋ɺ͔֬·ͩ޷ௐͷΑ͏Ͱ ୈ൛͕ग़Δͱ͔Ͱͳ͍ͱ͔ )BEPPQΛ׆༻ͨ͠ϓϩμΫτ ։ൃ

Slide 13

Slide 13 text

೥ w 4QBSL͕"QBDIF4PGUXBSF'PVOEBUJPO "4' ʹدଃ͞Εɺ"QBDIF4QBSLͱͳΔ w %BUBCSJDLT͕ελʔτ w ೔ຊͰ͸4QBSL͸·ͩ·ͩແ໊ͩͬͨ͸ͣ

Slide 14

Slide 14 text

w ͕ɺͲͪΒ͔ͱݴ͏ͱʮूܭʯͳײ͡ w େن໛σʔλʹର͢Δػցֶशͦͷଞͷχʔζ͕ ͦ͜·Ͱͳ͍ʁ w 8FCܥاۀͷϩάղੳͱ͔ϚʔέςΟϯά σʔλͷ෼ੳͱ͔ͦͷล͘Β͍ʁ w ͋·Γࣾ֎ʹཔΜͰ΍Δέʔε͸ଟ͘ͳͦ͞͏ େن໛σʔλॲཧͱݴ͑͹ )BEPPQͱ͍͏࣌୅ʹಥೖ

Slide 15

Slide 15 text

೥ w ࢲͱ4QBSL͕ຊ֨తʹग़ձͬͨ೥ w ৭ʑ͋ͬͯ4QBSLͰϏδωεΛ΍͍ͬͯΔΠϯ υاۀ΁ͷࢹ࡯஄ؙग़ுʹ͍͖·ͨ͠ w ׂͱૣΊʹඈͼͭ͘Ϣʔβʔ͕ग़࢝Ί͖ͯͨ͘ Β͍ʁ

Slide 16

Slide 16 text

4QBSLͷୈҰҹ৅ w 1ZUIPO΍3ͷΠϯλʔϑΣʔε͕͋Δʂ͢͝ ͍ʂ w ϦΞϧλΠϜॲཧ΋ಉ͡Α͏ʹॻ͍࣮ͯݱͰ͖Δ ͩͬͯʂ͍͢͝ʂ w ΤίγεςϜ͕Α͘ߟ͑ΒΕ͍ͯΔʂ͍͢͝ʂ

Slide 17

Slide 17 text

IUUQTPHJSBSEPUXPSEQSFTTDPNSEETBSFUIFOFXCZUFDPEFPGBQBDIFTQBSL

Slide 18

Slide 18 text

σʔλαΠΤϯςΟετ΋Τ ϯδχΞ΋๊͍͑ͯΔฐࣾͷ Α͏ͳاۀ΋औΓ૊Ή΂͖ʂ

Slide 19

Slide 19 text

೥ʹ޲͖߹ͬͨݱ࣮ w ϒϨΠϯύουࣾͱͯ͠ɼϏδωεϞσϧతʹ 4QBSLΛ࠾༻͍ͯ͘͠ϝϦοτ͕େ͖͘ͳ͍ w σʔλ෼ੳΛ౰֘اۀͷ֎෦Ͱҕୗ͞Εͯߦ ͏ձࣾͷ໋॓ͷ༷ͳ࿩ɾɾɾ w ͦͷҰํͰɺࣄۀձࣾͷํʑ͕ͲΜͲΜࣄྫ Λग़ͯ͘͠Δɾɾɾ͏Β΍·͍͠ɾɾɾ

Slide 20

Slide 20 text

ʢ࠶ܝʣڞஶऀͷํʑͷ෼ྨ w ࣄۀձࣾ w ϦΫϧʔτɺ%..DPNɺ4NBSU/FXT w ϕϯμʔʢϓϥοτϑΥʔϚʔʣ w )PSUPO8PSLT *#. w ϕϯμʔʢίϯαϧɺ4*ܥʣ w ϒϨΠϯύου

Slide 21

Slide 21 text

Կ͔৽͍ٕ͠ज़ ࣄۀձࣾ ϓϥοτϑΥʔϚʔ ίϯαϧ4* ϏδωεͱηοτͰߟ͑Δͱ ͨͿΜ͜Μͳײ͡Ͱ͢ ৽͍ٕ͠ज़͕ඞཁͱ ͞ΕΔ·ͰҰ൪ԕ͍

Slide 22

Slide 22 text

ϒϨΠϯύουͰ௚໘࣮ͨ͠৘ w ͦ΋ͦ΋ΫϥΠΞϯτʹ4QBSLͷΑ͏ͳϓϩμΫτΛ࠾ ༻͢Δχʔζ͕ඞཁ w 4QBSLͷಋೖ͔Βʢσʔλ෼ੳج൫ߏஙʣ෼ੳ·Ͱɺ ͱ͍͏χʔζΛ͍࣋ͬͯΔͷ͸΄ΜͷҰѲΓ w 4QBSLͷϝϦοτ͕ग़ͯ͘Δͷ͸શମͷΦϖϨʔγϣϯ ͱηοτͰ͋Γɺ੾Γग़͞ΕΔ෼ੳ෦෼ͷΈͰ͸30*త ͳԸܙ͕গͳͯ͘ఏҊ͢ΔϝϦοτͱͱͯ͠΋ɺࣗࣾ಺ ͷ෼ੳͷج൫ͱͯ͠ϝϦοτͱͱͯ͠΋ऑ͍ɾɾɾPS[

Slide 23

Slide 23 text

ͦ͜Ͱࢲୡ͸Ͳ͏͔ͨ͠ʁ w %FMUB$VCFͱ͍͏ࣗࣾ։ൃͷϓϩμΫτͰ࠾༻ ͠·ͨ͠ w ࣗࣾϓϩμΫτ΋͍࣋ͬͯΔͷ͕ٹ͍ͩͬͨ w ͜ͷลͷ࿩͸͜ͷޙɺฐࣾࢣԬͷύʔτͰ࿩ ͕Ͱ͖Ε͹ͱ

Slide 24

Slide 24 text

༨ஊɿਂ૚ֶशͷొ৔ w ਂ૚ֶशͷ౸དྷʹΑͬͯ·ͨগ͠෼ੳϏδωε ք۾ͷτϨϯυ͕มΘΓͭͭ͋Γ·͢ w ʮૣ͘ॲཧ͕Ͱ͖ͨํ͕ྑ͍ʯ͔Βʮ͕͔࣌ؒ ͔ͬͯ΋ྑ͍ʢͱ͍͏͔͔͔Δͷ͸࢓ํͳ͍ʣ ͷͰ͔ͬ͠Γͱͨ͠Ϟσϧ͕࡞ΒΕΔ͜ͱ͕ॏ ཁʯ

Slide 25

Slide 25 text

ਓͰ΋͋Δఔ౓Ͱ͖͍ͯΔͱ͜ΖΛػցతʹ΍Δ͜ͱͰ ޮ཰ԽΛਤΔ 30*ޮՌΛదਖ਼ʹଌΕΔΑ͏ʹ౤ࢿΛ͍ͨ͠ ෼ੳػցֶशΛத৺ʹ͜Ε·Ͱશ͘ߟ͑΋͔ͭͳ͔ͬͨ ΠϊϕʔγϣϯΛى͍ͨ͜͠ʢىͤ͜ΔͰ͠ΐʁʣ ๻ΒͷॴʹདྷΔ૬ஊ ͪΐͬͱ͸4QBSLͳͲΛఏҊ͠΍͍͢؀ڥʹͳ͖ͬͯͨʁ

Slide 26

Slide 26 text

·ͱΊ w ݱঢ়͸ɺϏδωεͱ͓ͯ͠࢓ࣄͰ4QBSLΛΰϦΰ Ϧ৮Δػձ͕࣋ͪʹ͍͘ w ͜ΕΛଧഁ͢Δʹ͸ɺ΋ͬͱ4QBSLͷ෼ੳ؀ڥ Λඞཁͱ͠ಋೖΛݕ౼͍ͯͨ͠Γɺ࣮ࡍʹ࣋ͬ ͍ͯͯ׆༻ʹࠔ͍ͬͯΔΑ͏ͳاۀ͕ඞཁ w ΋ͬͱ4QBSLΑྲྀߦͬͯ͘Εʂ

Slide 27

Slide 27 text

$IBOHFUP.PSPPLBTBO

Slide 28

Slide 28 text

σʔλϚωδϝϯτπʔϧ %FMUB$VCFͰͷ4QBSLར༻ ࢣԬҰ੒

Slide 29

Slide 29 text

ࣗݾ঺հ w גࣜձࣾϒϨΠϯύουͰΤϯδχΞͯ͠·͢ w ݩʑ͸)BEPPQ΍Γͨͯ͘ೖࣾ͠·͕ͨ͠ɺ͍ Ζ͍Ζ΍ͬͯ·͢ w ࠷ۙ͸ͳ͔ͥ0QFO*%$POOFDU࢖ͬͨೝূαʔ ό࡞ͬͯ·͢

Slide 30

Slide 30 text

4QBSLͱͷग़ձ͍ w ܰ͘৮ͬͨͷ͸೥͔Βɺຊ֨తʹۀ຿Ͱ࢖ ͍ͩͨ͠ͷ͸೥͔Β w ͦͷੲʹ)BEPPQͷ/BUJWF"1*ͰͷΞϓϦέʔγϣ ϯΛ։ൃɾӡ༻ͯͨ͠ͷͰɺ1ZUIPO࢖͑ͨΓ݁ ߹͕؆୯ʹग़དྷͨΓͰɺʮ͓͓ʔʯͬͯͳΔ

Slide 31

Slide 31 text

ຊ೔࿩͢͜ͱ w %FMUB$VCFͱ͍͏ϓϩμΫτͰ4QBSLΛಋೖ͠ ͨ࿩Λ͠·͢

Slide 32

Slide 32 text

%FMUB$VCF w ϓϥΠϕʔτ%.13UPBTUFSͰอ͍࣋ͯ͠Δ ߦಈϩά౳͔ΒηάϝϯτΛ࡞੒͢Δπʔϧ

Slide 33

Slide 33 text

%FMUB$VCF w ։ൃ͸೥͔Β։࢝ɺਖ਼ࣜϦϦʔε͸೥ w ։ൃ։͔࢝Β೥͘Β͍Ͱ͕͢ɺ͍Ζ͍ΖͳมભΛ ؚΊͯ࿩͍ͯ͜͠͏͔ͱࢥ͍·͢

Slide 34

Slide 34 text

ॳظɿ೥ʙ w ߦಈϩάΛखಈͰݕࡧͯ͠ηάϝϯτΛ࡞੒ w ηάϝϯτಉ࢜Λֻ͚͋ΘͤͨΓͱ͔৭ʑ

Slide 35

Slide 35 text

ॳظɿ೥ʙ w "84্ʹ1SFTUPΫϥελΛ૊ΜͰੜϩάݕࡧ w σʔλαΠζతʹ͸਺5# Web Servers Elastic Load Balancing Presto Cluster Coordinator Workers

Slide 36

Slide 36 text

தظɿ೥ʙ w ػೳతʹϚχΞοΫ࿏ઢ w ޿ࠂӡ༻ͷख͕͔͔ؒΒͳ͍Α͏ͳϥΠτͳײ ͡ͷ࿏ઢ͕ཁ๬ͱͯ͠ଟʑ͋Γ

Slide 37

Slide 37 text

தظɿ೥ʙ w ͦ͜ͰҰ࿈ͷྲྀΕΛࣗಈԽͯ͠Έͨ w ͜ͷϢʔβͷΫϥελϦϯάʹ4QBSLΛ࢖༻ ࣗಈͰϢʔβΛ ΫϥελϦϯά ࣗಈͰ"#ςετ ͯ͠࠷దԽ

Slide 38

Slide 38 text

தظɿ೥ʙ w &.3ͷ:"3/্Ͱ4QBSL w &5-ɾूܭɿ4QBSL$PSFɺ4QBSL42- w ΫϥελϦϯάɿ.-MJC Amazon EMR Web Servers Elastic Load Balancing Presto Cluster Coordinator Workers

Slide 39

Slide 39 text

தظɿ೥ʙ w ͳͥ4QBSLΛબΜ͔ͩʁ w ͸͍ɺ൒͹ແཧ໼ཧͰ͢ w ͨͩɺಋೖ͠ऴΘͬͯΈΔͱɺ&5-ɾूܭɾػցֶ श͕ಉҰͷϓϥοτϑΥʔϜɾݴޠɾϑϨʔϜϫʔ ΫͰ࣮૷Ͱ͖ͨͷ͸େ͖͔ͬͨͰ͢ w ։ൃظؒ΋ݕূؚΊͯϲ݄ఔ౓

Slide 40

Slide 40 text

ݱࡏ೥ʙ w :"3/ͱ1SFTUPͰΫϥελͭ͋ΔͷΠϠ w 1SFTUPແͯ͘͠4QBSL42-ʢ+%#$ʣʁ Amazon EMR Web Servers Elastic Load Balancing Presto Cluster Coordinator Workers

Slide 41

Slide 41 text

ݱࡏ೥ʙ w 1SFTUP͸ແ͍ͤͯ͘·ͤΜɻɻɻ w 4QBSL͸γϟοϑϧ࣌ʹσΟεΫॻࠐ͕͋ΔͷͰɺ ΦϯϥΠϯ͔ΒͷݕࡧੑೳͰ1SFTUPʹ܉഑

Slide 42

Slide 42 text

ݱࡏ೥ʙ w ࠷ऴతʹ͸&.3্ʹ1SFTUPΫϥελΛҠಈͤ͞ ͯӡ༻ͯ͠·͢ Web Servers Elastic Load Balancing Amazon EMR ΦϯϥΠϯܥ όονܥ

Slide 43

Slide 43 text

·ͱΊ w %FMUB$VCFͱ͍͏ϓϩμΫτͰ4QBSLΛಋೖͯ͠Έ ͨ࿩Λ͠·ͨ͠ w 4QBSL͸৭ʑͳίϯϙʔωϯτ͕ඪ४Ͱೖ͍ͬͯͯɺ .BQ3FEVDFʹൺ΂૬౰ෑډ͕௿͘ͳ͍ͬͯ·͢ w ΋ͬͱ4QBSLΑྲྀߦͬͯ͘Εʂ