1日100個以上のHadoopクラスターを使い捨てる方法 / How to throw away 100 Hadoop clusters a day

Transcript

1. ೥݄೔
   ιϑτόϯΫגࣜձࣾதཬߒ೭
   ೔ݸҎ্ͷ
   )BEPPQΫϥελʔΛ
   ࢖͍ࣺͯΔํ๏
   )BEPPQ4QBSL$POGFSFODF+BQBO
   

   View Slide

2. ࣗݾ঺հ
   
   
   தཬߒ೭ φΧβτώϩϢΩ
   
   ॴଐ
   ιϑτόϯΫגࣜձࣾ
   גࣜձࣾ"HPPQ
   ϙδγϣϯ
   όοΫΤϯυνʔϜϦʔμʔ
   "845FDIOJDBM.FJTUFS
   

   View Slide

3. 
   
   ͜ΜͳهࣄΛॻ͍͍ͯ·͢
   IUUQTRJJUBDPNOBLB[BYJUFNTFEEGBB
   

   View Slide

4. Ґஔ৘ใऩू ղੳ ਓͷྲྀΕ
   

   View Slide

5. 
   
   .VTU೔෼ͷϩάΛ໷ؒόον։࢝·Ͱʹ%8)ʹ౤ೖ
   8BOU%8)΁ͷ౤ೖ͸ૣ͚Ε͹ૣ͍΄Ͳϕλʔ
   ػೳ໘
   ඇػೳ໘
   ϩάͷϘϦϡʔϜ͸೔਺ԯ݅5#Φʔμʔ
   ࠓޙ΋ϩάྔ͕૿͑ଓ͚ΔͨΊεέʔϥϏϦςΟ͕ඞཁ
   ίετޮ཰ٴͼ҆ఆੑ͕ߴ͍͜ͱ
   &5-ॲཧͷཁ݅
   

   View Slide

6. 
   
   ϩάऩू
   γεςϜ
   4 3FETIJGU
   ,JOFTJT
   +FOLJOTPO&$
   %FWFMPQFST
   %BUB"OBMZTUT
   &-# &$
   ऩूɾ஝ੵ &5- ղੳ
   &$
   &$
   ':ΞʔΩςΫνϟʔ
   

   View Slide

7. 
   
   ϩάऩू
   γεςϜ
   4 3FETIJGU
   ,JOFTJT
   +FOLJOTPO&$
   %FWFMPQFST
   %BUB"OBMZTUT
   &-# &$
   ऩूɾ஝ੵ &5- ղੳ
   &$
   &$
   ':ΞʔΩςΫνϟʔ
   εέʔϥϏϦςΟ
   ͳͲʹ՝୊
   

   View Slide

8. 
   
   ࣌ؒ෼ͷϩάΛೖྗɺ1ZUIPONVMUJQSPDFTTJOHͰ
   $16Λ࢖ͬͯ%8)ʹϩʔυՄೳͳܗࣜʹม׵
   ୆͋Δͷ͸ର৅ͷϩά͕एׯҟͳΔͨΊ
   ૿͑ଓ͚Δϩάʹରͯ͠
   &$ͷεέʔϧΞοϓͰରԠ
   ౰࣌ͷ&5-ॲཧͷ࣮૷
   

   View Slide

9. 
   
   DYMBSHF
   ೥ϦϦʔε
   W$16(J#.FN
   ֹ݄໿ສԁ
   DYMBSHF
   ೥Լ०ϦϦʔε
   W$16(J#.FN
   ֹ݄໿ສԁ
   εέʔϧΞοϓ
   ಄ଧͪϦεΫ
   ίϯϐϡʔςΟϯά࠷దԽΠϯελϯελΠϓ
   

   View Slide

10. 
    
    
    
    ࣌୆&5-ॲཧ
    
    
    ࣌୆&5-ॲཧ
    ͋Δ࣌ؒͷॲཧ͕௕Ҿ͘ͱޙଓ͕٧·Δ
    &$ 4
    ࣌୆&5-ॲཧ
    &5-ॲཧΠϝʔδ
    

    View Slide

11. 
    
    ϩάऩू
    γεςϜ
    4 3FETIJGU
    ,JOFTJT
    +FOLJOTPO&$
    %FWFMPQFST
    %BUB"OBMZTUT
    &-# &$
    ऩूɾ஝ੵ &5- ղੳ
    &$
    &$
    ':ΞʔΩςΫνϟʔ
    ՝୊ͷղܾʹணख
    

    View Slide

12. 
    
    ৽ΞʔΩςΫνϟʔߏ૝
    εέʔϧΞ΢τϞσϧ
    4QBSLPO&.3Λ࠾༻
    ॲཧͷಠཱੑ
    ࣌ؒ෼ͷ&5-ॲཧΛ
    Ϋϥελʔ͕୲౰
    ࢖͍ࣺͯՄೳͳ
    &.3όονॲཧج൫
    

    View Slide

13. 
    
    ϩάऩू
    γεςϜ
    4
    3FETIJGU
    +FOLJOTPO&$
    %FWFMPQFST
    %BUB"OBMZTUT
    &5- ղੳ
    ऩूɾ஝ੵ
    ʜ
    ʜ
    4QBSLPO&.3
    )JWFPO&.3
    ʜ
    4
    -BNCEB
    ,JOFTJT
    &-# &$
    ৽ΞʔΩςΫνϟʔ
    (MVF
    

    View Slide

14. 
    
    ৽ΞʔΩςΫνϟʔͷϙΠϯτ
    &.3ͷεςοϓػೳͷ׆༻
    -BNCEBʹΑΔ&.3Ϋϥελʔͷ੍ޚ
    ϑϧϚωʔδυ)JWFϝλετΞͷ׆༻
    ΞυϗοΫ෼ੳ༻&.3Ϋϥελʔͷఏڙ
    

    View Slide

15. 
    
    &.3ͷεςοϓػೳͷ׆༻ 1 2 3 4
    εςοϓ೚ҙͷॲཧΛॱ൪ʹ࣮ߦ͢Δ࢓૊Έ
    શεςοϓ׬ྃޙʹΫϥελʔΛऴྃ͢Δ౳ͷ੍ޚ͕
    ؆୯ʹߦ͑Δ
    ৚݅෼ذ΍܁Γฦ͕͠ͳ͍γϯϓϧͳόονॲཧϑϩʔ
    Ͱ͋Ε͹εςοϓػೳͰे෼ΧόʔՄೳ
    

    View Slide

16. 
    
    &5-ॲཧͷεςοϓ
    &.3
    ຊ൪ࢿࡐ
    +BS
    4IFMM
    ςετࢿࡐ
    4IFMM
    +BS
    ੜϩά
    1BSRVFU
    &5-ॲཧ༻ͷࢿࡐΛ4͔Βμ΢ϯϩʔυ
    TEJTUDQͰॲཧର৅ͷϩάΛू໿
    TQBSLTVCNJUͰσʔλՃ޻ͱ1BSRVFUม׵Λ࣮ࢪ
    1 2
    3
    

    View Slide

17. 
    
    -BNCEBʹΑΔΫϥελʔ੍ޚ 1 2 3 4
    &.3Ϋϥελʔىಈ༻ͷ-BNCEBؔ਺Λ࣮૷
    1ZUIPOͰ3VO+PC'MPXΛίʔϧ
    3VO+PC'MPX͸ύϥϝʔλʔ͕ඇৗʹଟ͍
    ઃఆϑΝΠϧϥΠϒϥϦ)0$0/Ͱ؅ཧ͠࠶ར༻ੑΛ޲্
    1MBZ'SBNFXPSLͳͲͰ༗໊ͳ-JHIUCFOE͕ࣾ։ൃ
    ϩάऩूγεςϜͳͲͰ1MBZ'SBNFXPSLΛ׆༻ͯ͠
    ͓Γ)0$0/ʹ׳Ε͍ͯͨͷ͕࠾༻ཧ༝
    

    View Slide

18. 
    
    &5-༻Ϋϥελʔىಈϑϩʔ 1 2 3 4
    Name = hourly-etl
    ReleaseLabel = emr-5.20.0
    Steps = [
    {
    spark-submit
    hourly-etl.jar
    ${YYYYMMDD}/${HH}
    }
    ...
    IPVSMZDPOG
    
    IPVSMZDPOG
    )0$0/ύʔε
    3VO+PC'MPX࣮ߦ
    

    View Slide

19. 
    
    &5-ॲཧͷϦΧόϦʔ
    ੒ޭ ੒ޭ ࣦഊ
    ͕ҟৗऴྃ
    1 2 3 4
    ϦΧόϦ༻
    ىಈ༻
    Ϋϥελʔͷঢ়ଶΛ
    ఆظతʹνΣοΫ
    ىಈ༻-BNCEBΛ
    ݺͼग़͠
    ࣦഊͨ࣌ؒ͠ͷ
    ΫϥελʔΛىಈ
    

    View Slide

20. 
    
    ϑϧϚωʔδυ)JWFϝλετΞ 1 2 3 4
    "84(MVF%BUB$BUBMPH
    <&.3ରԠ>
    Ҏલ͸.Z42-PO3%4Λ)JWFϝλετΞͱͯ͠ར༻
    3%4͕410'ʹͳΒͳ͍Α͏ʹؾΛݣ͏ඞཁ͕͋ͬͨ
    ΫϥελʔΛେྔʹىಈ͢Δͱ઀ଓͰ͖ͳ͍FUD
    (MVF%BUB$BUBMPH͸ϑϧϚωʔδυ͔ͭಉ࣌઀ଓ਺੍ݶͳ͠
    

    View Slide

21. 
    
    ΞυϗοΫ෼ੳ༻Ϋϥελʔ
    ΤϯδχΞ΍ΞφϦετ͕
    ؆୯ʹ&.3ΫϥελʔΛىಈՄೳʹ
    1 2 3 4
    

    View Slide

22. 
    
    Ϋϥελʔͷམͱ͠๨Ε๷ࢭ 1 2 3 4
    I I I
    ؂ࢹ༻
    Ϋϥελʔͷ
    Քಇ࣌ؒΛνΣοΫ
    4MBDL௨஌
    BEIPDDMVTUFS
    
    

    View Slide

23. 
    
    ίετͷ࿩
    ຖ࣌ͷ&5-ॲཧ͸ҎલΑΓ҆Ձʹ
    εϙοτΠϯελϯεΛੵۃతʹ׆༻͍ͯ͠ΔͨΊ
    4શମͷίετ͸૿Ճ܏޲
    ΞυϗοΫ෼ੳ͕ؾܰʹͰ͖ΔͨΊ෼ੳ݁Ռͷσʔλ͕
    4ʹ஝ੵ͍͍ͯͬͯ͠Δ
    ෆཁͳσʔλΛఆظతʹ࡟আ͢ΔΑ͏ͳ࢓૊Έ͕ඞཁ
    

    View Slide

24. 
    
    ϩάऩू
    γεςϜ
    4
    3FETIJGU
    +FOLJOTPO&$
    %FWFMPQFST
    %BUB"OBMZTUT
    &5- ղੳ
    ऩूɾ஝ੵ
    ʜ
    ʜ
    4QBSLPO&.3
    )JWFPO&.3
    ʜ
    4
    -BNCEB
    ,JOFTJT
    &-# &$
    ࠶ܝ৽ΞʔΩςΫνϟʔ
    (MVF
    

    View Slide

25. THANK YOU!
    ANY QUESTIONS?
    

    View Slide