Upgrade to Pro — share decks privately, control downloads, hide ads and more …

1日100個以上のHadoopクラスターを使い捨てる方法 / How to throw away 100 Hadoop clusters a day

1日100個以上のHadoopクラスターを使い捨てる方法 / How to throw away 100 Hadoop clusters a day

More Decks by Hiroyuki Nakazato / 中里 浩之

Other Decks in Technology

Transcript

  1. ೥݄೔
    ιϑτόϯΫגࣜձࣾதཬߒ೭
    ೔ݸҎ্ͷ
    )BEPPQΫϥελʔΛ
    ࢖͍ࣺͯΔํ๏
    )BEPPQ4QBSL$POGFSFODF+BQBO

    View Slide

  2. ࣗݾ঺հ


    தཬߒ೭ φΧβτώϩϢΩ

    ॴଐ
    ιϑτόϯΫגࣜձࣾ
    גࣜձࣾ"HPPQ
    ϙδγϣϯ
    όοΫΤϯυνʔϜϦʔμʔ
    "845FDIOJDBM.FJTUFS

    View Slide



  3. ͜ΜͳهࣄΛॻ͍͍ͯ·͢
    IUUQTRJJUBDPNOBLB[BYJUFNTFEEGBB

    View Slide

  4. Ґஔ৘ใऩू ղੳ ਓͷྲྀΕ

    View Slide



  5. .VTU೔෼ͷϩάΛ໷ؒόον։࢝·Ͱʹ%8)ʹ౤ೖ
    8BOU%8)΁ͷ౤ೖ͸ૣ͚Ε͹ૣ͍΄Ͳϕλʔ
    ػೳ໘
    ඇػೳ໘
    ϩάͷϘϦϡʔϜ͸೔਺ԯ݅5#Φʔμʔ
    ࠓޙ΋ϩάྔ͕૿͑ଓ͚ΔͨΊεέʔϥϏϦςΟ͕ඞཁ
    ίετޮ཰ٴͼ҆ఆੑ͕ߴ͍͜ͱ
    &5-ॲཧͷཁ݅

    View Slide



  6. ϩάऩू
    γεςϜ
    4 3FETIJGU
    ,JOFTJT
    +FOLJOTPO&$
    %FWFMPQFST
    %BUB"OBMZTUT
    &-# &$
    ऩूɾ஝ੵ &5- ղੳ
    &$
    &$
    ':ΞʔΩςΫνϟʔ

    View Slide



  7. ϩάऩू
    γεςϜ
    4 3FETIJGU
    ,JOFTJT
    +FOLJOTPO&$
    %FWFMPQFST
    %BUB"OBMZTUT
    &-# &$
    ऩूɾ஝ੵ &5- ղੳ
    &$
    &$
    ':ΞʔΩςΫνϟʔ
    εέʔϥϏϦςΟ
    ͳͲʹ՝୊

    View Slide



  8. ࣌ؒ෼ͷϩάΛೖྗɺ1ZUIPONVMUJQSPDFTTJOHͰ
    $16Λ࢖ͬͯ%8)ʹϩʔυՄೳͳܗࣜʹม׵
    ୆͋Δͷ͸ର৅ͷϩά͕एׯҟͳΔͨΊ
    ૿͑ଓ͚Δϩάʹରͯ͠
    &$ͷεέʔϧΞοϓͰରԠ
    ౰࣌ͷ&5-ॲཧͷ࣮૷

    View Slide



  9. DYMBSHF
    ೥ϦϦʔε
    W$16(J#.FN
    ֹ݄໿ສԁ
    DYMBSHF
    ೥Լ०ϦϦʔε
    W$16(J#.FN
    ֹ݄໿ສԁ
    εέʔϧΞοϓ
    ಄ଧͪϦεΫ
    ίϯϐϡʔςΟϯά࠷దԽΠϯελϯελΠϓ

    View Slide





  10. ࣌୆&5-ॲཧ


    ࣌୆&5-ॲཧ
    ͋Δ࣌ؒͷॲཧ͕௕Ҿ͘ͱޙଓ͕٧·Δ
    &$ 4
    ࣌୆&5-ॲཧ
    &5-ॲཧΠϝʔδ

    View Slide



  11. ϩάऩू
    γεςϜ
    4 3FETIJGU
    ,JOFTJT
    +FOLJOTPO&$
    %FWFMPQFST
    %BUB"OBMZTUT
    &-# &$
    ऩूɾ஝ੵ &5- ղੳ
    &$
    &$
    ':ΞʔΩςΫνϟʔ
    ՝୊ͷղܾʹணख

    View Slide



  12. ৽ΞʔΩςΫνϟʔߏ૝
    εέʔϧΞ΢τϞσϧ
    4QBSLPO&.3Λ࠾༻
    ॲཧͷಠཱੑ
    ࣌ؒ෼ͷ&5-ॲཧΛ
    Ϋϥελʔ͕୲౰
    ࢖͍ࣺͯՄೳͳ
    &.3όονॲཧج൫

    View Slide



  13. ϩάऩू
    γεςϜ
    4
    3FETIJGU
    +FOLJOTPO&$
    %FWFMPQFST
    %BUB"OBMZTUT
    &5- ղੳ
    ऩूɾ஝ੵ
    ʜ
    ʜ
    4QBSLPO&.3
    )JWFPO&.3
    ʜ
    4
    -BNCEB
    ,JOFTJT
    &-# &$
    ৽ΞʔΩςΫνϟʔ
    (MVF

    View Slide



  14. ৽ΞʔΩςΫνϟʔͷϙΠϯτ
    &.3ͷεςοϓػೳͷ׆༻
    -BNCEBʹΑΔ&.3Ϋϥελʔͷ੍ޚ
    ϑϧϚωʔδυ)JWFϝλετΞͷ׆༻
    ΞυϗοΫ෼ੳ༻&.3Ϋϥελʔͷఏڙ

    View Slide



  15. &.3ͷεςοϓػೳͷ׆༻ 1 2 3 4
    εςοϓ೚ҙͷॲཧΛॱ൪ʹ࣮ߦ͢Δ࢓૊Έ
    શεςοϓ׬ྃޙʹΫϥελʔΛऴྃ͢Δ౳ͷ੍ޚ͕
    ؆୯ʹߦ͑Δ
    ৚݅෼ذ΍܁Γฦ͕͠ͳ͍γϯϓϧͳόονॲཧϑϩʔ
    Ͱ͋Ε͹εςοϓػೳͰे෼ΧόʔՄೳ

    View Slide



  16. &5-ॲཧͷεςοϓ
    &.3
    ຊ൪ࢿࡐ
    +BS
    4IFMM
    ςετࢿࡐ
    4IFMM
    +BS
    ੜϩά
    1BSRVFU
    &5-ॲཧ༻ͷࢿࡐΛ4͔Βμ΢ϯϩʔυ
    TEJTUDQͰॲཧର৅ͷϩάΛू໿
    TQBSLTVCNJUͰσʔλՃ޻ͱ1BSRVFUม׵Λ࣮ࢪ
    1 2
    3

    View Slide



  17. -BNCEBʹΑΔΫϥελʔ੍ޚ 1 2 3 4
    &.3Ϋϥελʔىಈ༻ͷ-BNCEBؔ਺Λ࣮૷
    1ZUIPOͰ3VO+PC'MPXΛίʔϧ
    3VO+PC'MPX͸ύϥϝʔλʔ͕ඇৗʹଟ͍
    ઃఆϑΝΠϧϥΠϒϥϦ)0$0/Ͱ؅ཧ͠࠶ར༻ੑΛ޲্
    1MBZ'SBNFXPSLͳͲͰ༗໊ͳ-JHIUCFOE͕ࣾ։ൃ
    ϩάऩूγεςϜͳͲͰ1MBZ'SBNFXPSLΛ׆༻ͯ͠
    ͓Γ)0$0/ʹ׳Ε͍ͯͨͷ͕࠾༻ཧ༝

    View Slide



  18. &5-༻Ϋϥελʔىಈϑϩʔ 1 2 3 4
    Name = hourly-etl
    ReleaseLabel = emr-5.20.0
    Steps = [
    {
    spark-submit
    hourly-etl.jar
    ${YYYYMMDD}/${HH}
    }
    ...
    IPVSMZDPOG

    IPVSMZDPOG
    )0$0/ύʔε
    3VO+PC'MPX࣮ߦ

    View Slide



  19. &5-ॲཧͷϦΧόϦʔ
    ੒ޭ ੒ޭ ࣦഊ
    ͕ҟৗऴྃ
    1 2 3 4
    ϦΧόϦ༻
    ىಈ༻
    Ϋϥελʔͷঢ়ଶΛ
    ఆظతʹνΣοΫ
    ىಈ༻-BNCEBΛ
    ݺͼग़͠
    ࣦഊͨ࣌ؒ͠ͷ
    ΫϥελʔΛىಈ

    View Slide



  20. ϑϧϚωʔδυ)JWFϝλετΞ 1 2 3 4
    "84(MVF%BUB$BUBMPH
    <&.3ରԠ>
    Ҏલ͸.Z42-PO3%4Λ)JWFϝλετΞͱͯ͠ར༻
    3%4͕410'ʹͳΒͳ͍Α͏ʹؾΛݣ͏ඞཁ͕͋ͬͨ
    ΫϥελʔΛେྔʹىಈ͢Δͱ઀ଓͰ͖ͳ͍FUD
    (MVF%BUB$BUBMPH͸ϑϧϚωʔδυ͔ͭಉ࣌઀ଓ਺੍ݶͳ͠

    View Slide



  21. ΞυϗοΫ෼ੳ༻Ϋϥελʔ
    ΤϯδχΞ΍ΞφϦετ͕
    ؆୯ʹ&.3ΫϥελʔΛىಈՄೳʹ
    1 2 3 4

    View Slide



  22. Ϋϥελʔͷམͱ͠๨Ε๷ࢭ 1 2 3 4
    I I I
    ؂ࢹ༻
    Ϋϥελʔͷ
    Քಇ࣌ؒΛνΣοΫ
    4MBDL௨஌
    BEIPDDMVTUFS

    View Slide



  23. ίετͷ࿩
    ຖ࣌ͷ&5-ॲཧ͸ҎલΑΓ҆Ձʹ
    εϙοτΠϯελϯεΛੵۃతʹ׆༻͍ͯ͠ΔͨΊ
    4શମͷίετ͸૿Ճ܏޲
    ΞυϗοΫ෼ੳ͕ؾܰʹͰ͖ΔͨΊ෼ੳ݁Ռͷσʔλ͕
    4ʹ஝ੵ͍͍ͯͬͯ͠Δ
    ෆཁͳσʔλΛఆظతʹ࡟আ͢ΔΑ͏ͳ࢓૊Έ͕ඞཁ

    View Slide



  24. ϩάऩू
    γεςϜ
    4
    3FETIJGU
    +FOLJOTPO&$
    %FWFMPQFST
    %BUB"OBMZTUT
    &5- ղੳ
    ऩूɾ஝ੵ
    ʜ
    ʜ
    4QBSLPO&.3
    )JWFPO&.3
    ʜ
    4
    -BNCEB
    ,JOFTJT
    &-# &$
    ࠶ܝ৽ΞʔΩςΫνϟʔ
    (MVF

    View Slide

  25. THANK YOU!
    ANY QUESTIONS?

    View Slide