1日100個以上のHadoopクラスターを使い捨てる方法 / How to throw away 100 Hadoop clusters a day

Slide 1

Slide 1 text

೥݄೔ ιϑτόϯΫגࣜձࣾதཬߒ೭ ೔ݸҎ্ͷ )BEPPQΫϥελʔΛ ࢖͍ࣺͯΔํ๏ )BEPPQ4QBSL$POGFSFODF+BQBO

Slide 2

Slide 2 text

ࣗݾ঺հ தཬߒ೭ φΧβτώϩϢΩ ॴଐ ιϑτόϯΫגࣜձࣾ גࣜձࣾ"HPPQ ϙδγϣϯ όοΫΤϯυνʔϜϦʔμʔ "845FDIOJDBM.FJTUFS

Slide 3

Slide 3 text

͜ΜͳهࣄΛॻ͍͍ͯ·͢ IUUQTRJJUBDPNOBLB[BYJUFNTFEEGBB

Slide 4

Slide 4 text

Ґஔ৘ใऩू ղੳ ਓͷྲྀΕ

Slide 5

Slide 5 text

.VTU೔෼ͷϩάΛ໷ؒόον։࢝·Ͱʹ%8)ʹ౤ೖ 8BOU%8)΁ͷ౤ೖ͸ૣ͚Ε͹ૣ͍΄Ͳϕλʔ ػೳ໘ ඇػೳ໘ ϩάͷϘϦϡʔϜ͸೔਺ԯ݅5#Φʔμʔ ࠓޙ΋ϩάྔ͕૿͑ଓ͚ΔͨΊεέʔϥϏϦςΟ͕ඞཁ ίετޮ཰ٴͼ҆ఆੑ͕ߴ͍͜ͱ &5-ॲཧͷཁ݅

Slide 6

Slide 6 text

ϩάऩू γεςϜ 4 3FETIJGU ,JOFTJT +FOLJOTPO&$ %FWFMPQFST %BUB"OBMZTUT &-# &$ ऩूɾ஝ੵ &5- ղੳ &$ &$ ':ΞʔΩςΫνϟʔ

Slide 7

Slide 7 text

ϩάऩू γεςϜ 4 3FETIJGU ,JOFTJT +FOLJOTPO&$ %FWFMPQFST %BUB"OBMZTUT &-# &$ ऩूɾ஝ੵ &5- ղੳ &$ &$ ':ΞʔΩςΫνϟʔ εέʔϥϏϦςΟ ͳͲʹ՝୊

Slide 8

Slide 8 text

࣌ؒ෼ͷϩάΛೖྗɺ1ZUIPONVMUJQSPDFTTJOHͰ $16Λ࢖ͬͯ%8)ʹϩʔυՄೳͳܗࣜʹม׵ ୆͋Δͷ͸ର৅ͷϩά͕एׯҟͳΔͨΊ ૿͑ଓ͚Δϩάʹରͯ͠ &$ͷεέʔϧΞοϓͰରԠ ౰࣌ͷ&5-ॲཧͷ࣮૷

Slide 9

Slide 9 text

DYMBSHF ೥ϦϦʔε W$16(J#.FN ֹ݄໿ສԁ DYMBSHF ೥Լ०ϦϦʔε W$16(J#.FN ֹ݄໿ສԁ εέʔϧΞοϓ ಄ଧͪϦεΫ ίϯϐϡʔςΟϯά࠷దԽΠϯελϯελΠϓ

Slide 10

Slide 10 text

࣌୆&5-ॲཧ ࣌୆&5-ॲཧ ͋Δ࣌ؒͷॲཧ͕௕Ҿ͘ͱޙଓ͕٧·Δ &$ 4 ࣌୆&5-ॲཧ &5-ॲཧΠϝʔδ

Slide 11

Slide 11 text

ϩάऩू γεςϜ 4 3FETIJGU ,JOFTJT +FOLJOTPO&$ %FWFMPQFST %BUB"OBMZTUT &-# &$ ऩूɾ஝ੵ &5- ղੳ &$ &$ ':ΞʔΩςΫνϟʔ ՝୊ͷղܾʹணख

Slide 12

Slide 12 text

৽ΞʔΩςΫνϟʔߏ૝ εέʔϧΞ΢τϞσϧ 4QBSLPO&.3Λ࠾༻ ॲཧͷಠཱੑ ࣌ؒ෼ͷ&5-ॲཧΛ Ϋϥελʔ͕୲౰ ࢖͍ࣺͯՄೳͳ &.3όονॲཧج൫

Slide 13

Slide 13 text

ϩάऩू γεςϜ 4 3FETIJGU +FOLJOTPO&$ %FWFMPQFST %BUB"OBMZTUT &5- ղੳ ऩूɾ஝ੵ ʜ ʜ 4QBSLPO&.3 )JWFPO&.3 ʜ 4 -BNCEB ,JOFTJT &-# &$ ৽ΞʔΩςΫνϟʔ (MVF

Slide 14

Slide 14 text

৽ΞʔΩςΫνϟʔͷϙΠϯτ &.3ͷεςοϓػೳͷ׆༻ -BNCEBʹΑΔ&.3Ϋϥελʔͷ੍ޚ ϑϧϚωʔδυ)JWFϝλετΞͷ׆༻ ΞυϗοΫ෼ੳ༻&.3Ϋϥελʔͷఏڙ

Slide 15

Slide 15 text

&.3ͷεςοϓػೳͷ׆༻ 1 2 3 4 εςοϓ೚ҙͷॲཧΛॱ൪ʹ࣮ߦ͢Δ࢓૊Έ શεςοϓ׬ྃޙʹΫϥελʔΛऴྃ͢Δ౳ͷ੍ޚ͕ ؆୯ʹߦ͑Δ ৚݅෼ذ΍܁Γฦ͕͠ͳ͍γϯϓϧͳόονॲཧϑϩʔ Ͱ͋Ε͹εςοϓػೳͰे෼ΧόʔՄೳ

Slide 16

Slide 16 text

&5-ॲཧͷεςοϓ &.3 ຊ൪ࢿࡐ +BS 4IFMM ςετࢿࡐ 4IFMM +BS ੜϩά 1BSRVFU &5-ॲཧ༻ͷࢿࡐΛ4͔Βμ΢ϯϩʔυ TEJTUDQͰॲཧର৅ͷϩάΛू໿ TQBSLTVCNJUͰσʔλՃ޻ͱ1BSRVFUม׵Λ࣮ࢪ 1 2 3

Slide 17

Slide 17 text

-BNCEBʹΑΔΫϥελʔ੍ޚ 1 2 3 4 &.3Ϋϥελʔىಈ༻ͷ-BNCEBؔ਺Λ࣮૷ 1ZUIPOͰ3VO+PC'MPXΛίʔϧ 3VO+PC'MPX͸ύϥϝʔλʔ͕ඇৗʹଟ͍ ઃఆϑΝΠϧϥΠϒϥϦ)0$0/Ͱ؅ཧ͠࠶ར༻ੑΛ޲্ 1MBZ'SBNFXPSLͳͲͰ༗໊ͳ-JHIUCFOE͕ࣾ։ൃ ϩάऩूγεςϜͳͲͰ1MBZ'SBNFXPSLΛ׆༻ͯ͠ ͓Γ)0$0/ʹ׳Ε͍ͯͨͷ͕࠾༻ཧ༝

Slide 18

Slide 18 text

&5-༻Ϋϥελʔىಈϑϩʔ 1 2 3 4 Name = hourly-etl ReleaseLabel = emr-5.20.0 Steps = [ { spark-submit hourly-etl.jar ${YYYYMMDD}/${HH} } ... IPVSMZDPOG IPVSMZDPOG )0$0/ύʔε 3VO+PC'MPX࣮ߦ

Slide 19

Slide 19 text

&5-ॲཧͷϦΧόϦʔ ੒ޭ ੒ޭ ࣦഊ ͕ҟৗऴྃ 1 2 3 4 ϦΧόϦ༻ ىಈ༻ Ϋϥελʔͷঢ়ଶΛ ఆظతʹνΣοΫ ىಈ༻-BNCEBΛ ݺͼग़͠ ࣦഊͨ࣌ؒ͠ͷ ΫϥελʔΛىಈ

Slide 20

Slide 20 text

ϑϧϚωʔδυ)JWFϝλετΞ 1 2 3 4 "84(MVF%BUB$BUBMPH <&.3ରԠ> Ҏલ͸.Z42-PO3%4Λ)JWFϝλετΞͱͯ͠ར༻ 3%4͕410'ʹͳΒͳ͍Α͏ʹؾΛݣ͏ඞཁ͕͋ͬͨ ΫϥελʔΛେྔʹىಈ͢Δͱ઀ଓͰ͖ͳ͍FUD (MVF%BUB$BUBMPH͸ϑϧϚωʔδυ͔ͭಉ࣌઀ଓ਺੍ݶͳ͠

Slide 21

Slide 21 text

ΞυϗοΫ෼ੳ༻Ϋϥελʔ ΤϯδχΞ΍ΞφϦετ͕ ؆୯ʹ&.3ΫϥελʔΛىಈՄೳʹ 1 2 3 4

Slide 22

Slide 22 text

Ϋϥελʔͷམͱ͠๨Ε๷ࢭ 1 2 3 4 I I I ؂ࢹ༻ Ϋϥελʔͷ Քಇ࣌ؒΛνΣοΫ 4MBDL௨஌ BEIPDDMVTUFS

Slide 23

Slide 23 text

ίετͷ࿩ ຖ࣌ͷ&5-ॲཧ͸ҎલΑΓ҆Ձʹ εϙοτΠϯελϯεΛੵۃతʹ׆༻͍ͯ͠ΔͨΊ 4શମͷίετ͸૿Ճ܏޲ ΞυϗοΫ෼ੳ͕ؾܰʹͰ͖ΔͨΊ෼ੳ݁Ռͷσʔλ͕ 4ʹ஝ੵ͍͍ͯͬͯ͠Δ ෆཁͳσʔλΛఆظతʹ࡟আ͢ΔΑ͏ͳ࢓૊Έ͕ඞཁ

Slide 24

Slide 24 text

ϩάऩू γεςϜ 4 3FETIJGU +FOLJOTPO&$ %FWFMPQFST %BUB"OBMZTUT &5- ղੳ ऩूɾ஝ੵ ʜ ʜ 4QBSLPO&.3 )JWFPO&.3 ʜ 4 -BNCEB ,JOFTJT &-# &$ ࠶ܝ৽ΞʔΩςΫνϟʔ (MVF

Slide 25

Slide 25 text

THANK YOU! ANY QUESTIONS?