Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ペパボにおけるOpenStackHacks!

 ペパボにおけるOpenStackHacks!

GMO Pepabo.incのOpenStack運用事例です。

Kazuhiko Yamashita

July 04, 2015
Tweet

More Decks by Kazuhiko Yamashita

Other Decks in Technology

Transcript

  1. d๻ͱՆΛҰॹʹτΡΪϟβʔ͠·ઐՊʂʂฤd
    GMO Pepabo, Inc.
    Pࢁ
    2015/07/04 Pepabo Tech Conference in Fukuoka
    ϖύϘʹ͓͚Δ

    0QFO4UBDL)BDLT

    View Slide

  2. Ͳ΋ɺ΅͘Ͱ͢
    1ࢁ
    (.01FQBCP*OD
    !QZBNB
    UFOTOBQPODPN

    View Slide

  3. "HFOEB
    0QFO4UBDLJT
    /ZBIJT
    IBDLT
    ࠓ΍ͬͯΔ͜ͱ

    View Slide

  4. 0QFO4UBDLJT
    Ϋϥ΢υج൫
    7.Λ؅ཧ͢ΔͨΊͷ

    ιϑτ΢ΣΞ܈

    View Slide

  5. ߏ੒ਤ
    /PWB"1*
    )PSJ[PO
    ,FZTUPOF
    /PWB$PNQVUF
    ෺ཧϗετ

    /FVUSPO
    $JOEFS
    HMBODF
    ͜Μͳ7.࡞Γ͍ͨ
    ͜ͷϢʔβʔݖݶ
    ͋Δʁ
    ࢖༻͢ΔΠϝʔδ
    04͸͜Ε

    σΟεΫ͸͜͜Λ
    ࢖͏
    $16ɼϝϞϦ͸
    ͜Ε
    ࢖༻͢ΔωοτϫʔΫ͸͜Ε
    ΦϨΦϨʂԶͩΑʂ

    View Slide

  6. ϖύϘ͕ղܾͨ͠
    ͔ͬͨ͜ͱ

    View Slide

  7. ݸʑͷαʔϏεͰಠࣗʹΠϯϑϥΛߏங

    View Slide

  8. αʔόΛؾܰʹ࡞Γ͍ͨ
    ຐ๏ͷΑ͏ʹ؆୯ʹ
    αʔό࡞Γ͍ͨΘ͊ɻɻ

    View Slide

  9. “2014/05/01 Issueݐઃ
    αʔϏε͝ͱͰ͸ͳ͘ɺ

    શࣾڞ௨ͷΠϯϑϥج൫Λ࡞Δͧʂʂʂʂ

    View Slide

  10. /ZBI

    View Slide

  11. /ZBIJT
    ϖύϘͷԾ૝ج൫Πϯϑϥ
    ͜Ε·ͰαʔϏε͝ͱʹઃ͚͍ͯͨ

    ΠϯϑϥΛ0QFO4UBDLͰ౷߹

    View Slide

  12. /ZBIJT
    IUUQXXXTMJEFTIBSFOFUVNF@QCUDCPC
    IUUQTXXXZPVUVCFDPNXBUDI W(618.;CP1T6

    View Slide

  13. /ZBIͷ༝དྷ
    /PBIʹܾ·Δ
    தೋපͬΆ͍ͬͯͳΔ
    Ұจࣈม͑ͯ/ZBIʹ͠Α͏ͥͬͯͳΔ
    ޙ෇Ͱ͔͍͍ͬ͜౰ͯࣈߟ͑Α͏ͥͬͯͳΔ
    /ZBIJTZFUBOPUIFSIPTUJOH

    View Slide

  14. /ZBI
    αʔϏε͝ͱʹςφϯτΛ
    ෼ׂ͠ɺݖݶ΍

    ϦιʔεΛ؅ཧ

    View Slide

  15. ߏ੒؅ཧ

    View Slide


  16. http://techlife.cookpad.com/entry/2015/05/12/080000

    View Slide

  17. JUBNBF
    $JOEFS
    /PWB
    $PNQVUF
    /FVUSPO
    /PWB
    $PNQVUF
    /PWB
    $PNQVUF
    /PWB
    $PNQVUF
    ŠŠ
    ŠŠ
    ŠŠ
    ŠŠ
    ŠŠ
    ŠŠ
    ŠŠ
    ŠŠ
    ŠŠ
    (JUͰόʔδϣϯ؅ཧ
    OPWBDPOG
    ʜ
    OFVUSPODPOG
    ʜ
    DJOEFSDPOG
    ʜ
    TFSWJDFOPWBSFTUBSU
    TFSWJDFOFVUSPOSFTUBSU
    TFSWJDFDJOEFSSFTUBSU

    View Slide

  18. JUBNBF
    OPWBOPEF CBTFSPMF NBDLFSFMSFDJQF
    OPWBSPMF DPNQVUFSFDJQF
    BQJSFDJQF
    CVOEMFFYFDJUBNBFTTIIOPWBQCDMPVEDPNa
    ŠOPEFKTPOOPEFOPWBKTPOCPPUTUSBQSC

    View Slide

  19. JUBNBF
    TFSWJDFPQFOTUBDLOPWBDPNQVUF
    SFNPUF@pMFFUDOPWBOPWBDPOGEP
    OPUJpFTSFTUBSU TFSWJDF
    FOE

    View Slide

  20. JUBNBF
    TFSWJDFPQFOTUBDLOPWBDPNQVUF
    SFNPUF@pMFFUDOPWBOPWBDPOGEP
    OPUJpFTSFTUBSU TFSWJDF
    FOE

    View Slide

  21. ؂ࢹ

    View Slide

  22. .BDLFSFM
    ͸ͯͳ͕ࣾఏڙ͢Δαʔό

    ؂ࢹαʔϏε
    .BDLFSFMḉ
    ϩʔϧͰ؂ࢹઃఆΛଋͶΔ͜ͱ͕

    Մೳ

    View Slide

  23. .BDLFSFM
    NBDLFSFM\
    SPMFOZBIOFVUSPOOPEF
    DPOG\
    NPOJUPS<
    \
    OBNFQMVHJODIFDLTOFVUSPOTFSWFS
    DPNNBOEVTSMJCOBHJPTQMVHJOTDIFDL@QSPDTBOFVUSPOTFSWFSD
    ^
    \
    OBNFQMVHJODIFDLTOFVUSPO
    DPNNBOEVTSMJCOBHJPTQMVHJOTDIFDL@UDQ)MPDBMIPTUQXD
    ^
    ʜ
    >
    ^
    ^

    View Slide

  24. .BDLFSFM
    /FVUSPO
    ŠŠ
    ŠŠ
    ŠŠ
    JUBNBFNBDLSFMSC
    SPMFT<OZBIOFVUSPOOPEF>

    DPNNBOEVTSMJCOBHJPTQMVHJOTDIFDL@QSPDTBOFVUSPOTFSWFSD

    DPNNBOEVTSMJCOBHJPTQMVHJOTDIFDL@UDQ)MPDBMIPTUQXDz
    ʜ
    NBDLFSFMḉ
    ࢮ׆ɺ͖͍͠஋؂ࢹ

    View Slide

  25. .BDLFSFM

    View Slide

  26. .BDLFSFM
    ઃఆ͸ϗετଆ͚ͩͰ*1ΞυϨεؾʹ
    ͠ͳ͍͍ͯ͘
    ઃఆϑΝΠϧ͕γϯϓϧͰ؅ཧ͠΍͍͢
    ؂ࢹఆٛ΋(6*ͰαΫαΫ΍ΕΔ͔ͭɺ
    Ұ౓ೖΕͯ͠·͑͹͋ͱ͸ಉ͡ϩʔϧ
    ʹࢀՃͤ͞Δ͚ͩ

    View Slide

  27. /ZBI·ͱΊ
    0QFO4UBDL࢖ͬͯ·͢
    ϖύϘͷϓϥΠϕʔτΫϥ΢υ
    αʔϏε୯ҐͰϦιʔε؅ཧ
    ߏ੒؅ཧ͸JUBNBF
    ؂ࢹ͸NBDLFSFM

    View Slide

  28. )BDLT

    View Slide


  29. View Slide

  30. େମग़དྷͨ

    View Slide

  31. ͡Ό͋ɺ৮ͬͯΈͯ
    ͍͍͔͢ʂʂʂ

    View Slide

  32. )PSJ[PO
    7.؅ཧ
    /8؅ཧ
    ηΩϡϦςΟʔ؅ཧ
    Ϣʔβʔ؅ཧ
    Ϧιʔε؅ཧ

    View Slide

  33. 0QFO4UBDL$MJFOU
    1ZUIPOͷΫϥΠϯτ
    OPWBMJTU
    OFVUSPOQPSUMJTU
    LFZTUPOFFOEQPJOUMJTU
    IUUQEPDTPQFOTUBDLPSHKBVTFSHVJEFDPOUFOUJOTUBMM@DMJFOUTIUNM

    View Slide

  34. GNʜ
    ͡Ό͋ɺԿʁ
    ୆ϗετ࡞Δͷʹճ
    8*ૢ࡞ͨ͠ΓɺίϚϯυ౤
    ͛ͳ͖Ό͍͚ͳ͍Θ͚Ͱ͔͢ʁ

    View Slide

  35. ΍ͬͯΒΜͳ͍ʂʂʂ

    View Slide

  36. )BDLT
    LBOBNFϢʔβʔ؅ཧ
    LBLJOFηΩϡϦςΟάϧʔϓ؅ཧ
    OZBIDMJ7.࡞੒
    1FD7.T࡞੒

    View Slide

  37. Ϣʔβʔ؅ཧ

    View Slide

  38. LBOBNF
    ϖύϘͰ͸ਓͷΤϯδχΞ͕

    ෳ਺ͷαʔϏεʹܞΘΔ
    )PSJ[POͩͱมߋཤྺ͕؅ཧͰ͖ͳ͍

    ূ੻͕࢒Βͳ͍

    View Slide

  39. LBOBNF
    ઃఆϑΝΠϧΛZBNMͰఆٛ
    ઃఆϑΝΠϧΛ(JUʹू໿

    ʮ୭͕ɺ͍ͭɺͲͷݖݶʯΛอ͔࣋ͨ͠Λ؅ཧ
    QZBNB
    FNBJMlQZBNB!QFQBCPDPNl
    UFOBOUT
    NVVNVVlDJUZCPZ
    EFWFMPQlBENJOz
    UBOBLB
    FNBJMlUBOBLB!QFQBCPDPN
    UFOBOUT
    EFWFMPQNFOUNFNCFS

    View Slide

  40. LBOBNF
    ొ࿥಺༰
    ,FZTUPOF
    QZBNB
    FNBJMlQZBNB!QFQBCPDPNl
    UFOBOUT
    NVVNVVlDJUZCPZ
    EFWFMPQlBENJOz
    UBOBLB
    FNBJMlUBOBLB!QFQBCPDPN
    UFOBOUT
    EFWFMPQNFOUNFNCFS
    (JU)VC13
    ɹొ࿥಺༰ʹޡΓ͕ͳ͍͜ͱΛ

    ɹ֬ೝ͢Δ
    (JU)VCͰͷϨϏϡʔ׬ྃޙɺొ࿥

    View Slide

  41. ηΩϡϦςΟ؅ཧ

    View Slide

  42. DPNQVUFOPEF ෺ཧϗετ

    JOUFSGBDF
    4FDVSJUZ(SPVQ
    Ϝʔ7.
    /PWB"QJ
    /FVUSPO
    Ϝʔ7.ʹ͸ΦϑΟε͔Βͷ
    44)ڐՄ͍ͨ͠
    Ϝʔ7.ʹ44)ڐՄ
    JQUBCMFTఆٛ
    7.Ͱ͸ͳ͘DPNQVUFOPEFʹ
    JQUBCMFTఆٛ

    View Slide

  43. LBLJOF
    ઃఆϑΝΠϧΛZBNMͰఆٛ
    ઃఆϑΝΠϧΛ(JUʹू໿
    ηΩϡϦςΟઃఆͷϨϏϡʔ͕ՄೳʹͳΓɺ

    ·ͨɺొ࿥ɺ੾Γ໭͠ͷख͕ؒେ෯ʹ࡟ݮ
    JOUFSOBM@QFSNJU
    SVMFT
    EJSFDUJPOJOHSFTT
    QSPUPDPM
    FUIFSUZQF*1W
    QPSU
    SFNPUF@JQ
    EFTDSJQUJPOΠϯλʔφϧઢͷ௨৴ΛશͯڐՄ

    View Slide

  44. LBLJOF
    طଘઃఆͷऔಘɾฤूɾొ࿥
    /PWB"QJ
    JOUFSOBM@QFSNJU
    SVMFT
    EJSFDUJPOJOHSFTT
    QSPUPDPM
    FUIFSUZQF*1W
    QPSU
    SFNPUF@JQ
    EFTDSJQUJPOΠϯλʔφϧઢͷ௨৴ΛશͯڐՄ
    ొ࿥͞Εͨ಺༰Λ൓ө
    ࠷৽ొ࿥಺༰Λ(JU؅ཧ
    /PWB$PNQVUF
    /PWB$PNQVUF
    /PWB$PNQVUF

    View Slide

  45. Πϯελϯε࡞੒

    View Slide

  46. JNBHF

    View Slide

  47. JNBHF
    TVEPϢʔβʔͷ࡞੒
    TTIαʔόʔͷ࣮ߦ
    DMPVEJOJU PQFOTUBDLBQJDMJFOU QZUIPO
    ͷ

    Πϯετʔϧ
    σΟεΫ࠷খԽͱΠϯελϯεىಈ࣌ʹࣗಈ֦ு
    ωοτϫʔΫॳظԽ
    ϑΝΠΞ΢ΥʔϧͷແޮԽ
    ೔ຊ࣌ؒΛઃఆ
    ىಈ࣌ͷTMBDL௨஌

    View Slide

  48. JNBHF
    $FOU04 

    XJUIDIFGQVQQFU

    6CVOUV 

    XJUIDIFGQVQQFU

    $PSF04ͳͲͳͲ

    View Slide

  49. JOTUBODF
    JNBHF
    CBTF
    &TTFODF
    QVQQFU
    DIFG
    DMPVEJOJU

    View Slide

  50. “Πϝʔδͷىಈ࣌ʹ౉͞Εͨuser-dataʹج͖ͮ
    OSʹରͯ͠ઃఆΛߦ͑Δ࢓૊Έ
    $MPVE*OJU
    DMPVEJOJU DMPVEDPOpH
    ϦϑΝϨϯε
    IUUQT[FUUBJPOCIKFMQDPNNVOJUZBSUJLMFSPHPQQMBFSJOHDMPVEJOJUSFGFSFODF

    View Slide

  51. $MPVE*OJU
    IPTUOBNFQZBNBEFNP
    GREOQZBNBEFNPDPN
    SFQP@SFMFBTFWFS
    VTFST
    OBNFQZBNB
    HSPVQTQZBNB
    TIFMMCJOCBTI
    TVEP<"-- "--
    /01"448%"-->
    TTIBVUIPSJ[FELFZT
    TTISTB'%"%'"%#/[B$ZD&GEMGKBTE&"R6+WTW,H)3.)E
    CPPUDNE
    FDIPSPPUQZBNBcDIQBTTXE

    View Slide

  52. /ZBI$MJ
    (PͰ։ൃ͞Εͨ"1*ϥούʔίϚϯυ
    ίϚϯυҰൃͰ7.Λىಈ
    VTFSEBUBʹରԠ͍ͯ͠ΔͷͰ೚ҙͷઃఆ͕ىಈޙʹ

    ߦΘΕΔ
    *1ΞυϨεઃఆΛࣗಈԽ
    VE[VSBOZBIDMJ
    8SBQQFSPG/ZBIUIF8SBQQFSPG0QFO4UBDL
    OZBIJOTUBODFTUEFWOZBIa
    JDFOUPTOQZBNBUFTUQQZBNBQBTTa
    ŠOFUXPSLFYUOFUFYU JOUOZBIa
    GNTNBMMVTFSEBUBVTFSEBUBZNM

    View Slide

  53. 1FD
    ઃఆϑΝΠϧʹج͖ͮෳ਺ͷ7.Λ

    Ұׅىಈɾఀࢭ
    VTFSEBUBͷςϯϓϨʔτԽΛαϙʔτ
    QZBNBUFTU
    UFOBOUQFQBCP
    JNBHFDFOUPT@DIFG@QVQQFU
    qBWPSNTNBMM
    OFUXPSLT
    FUI
    CPPUQSPUPTUBUJD
    JQ@BEESFTT
    HBUFXBZ
    EOT
    TFDVSJUZ@HSPVQ
    EFGBVMU
    TTI
    UFNQMBUFT
    CBTFZBNM
    XFCTFSWFSZBNM

    View Slide

  54. )BDLT·ͱΊ
    Ϣʔβʔ؅ཧΛূ੻Λ࢒ͭͭ͠

    ָʹʂʂ
    ηΩϡϦςΟʔ؅ཧ΍Πϯελϯε
    ࡞੒ͷखؒΛίʔυͰղܾ

    View Slide

  55. ࠓ΍ͬͯΔ͜ͱ

    View Slide

  56. *1Λҙࣝ͠ͳ͍
    ΠϯϑϥΛίʔυͰ
    ࡞Γ͖Δ

    View Slide

  57. Α͋͘Δ%/4γεςϜ
    -# -#
    %/4 %/4 %/4
    WJQ
    SJQ SJQ
    SJQ
    SJQ
    SJQ

    View Slide

  58. “)BTIJ$PSQࣾͷϓϩμΫτ
    *(.1Λར༻͠ಉ͡άϧʔϓͷϝϯόʔͷ௥Ճ΍
    ཭୤ΛΠϕϯτ௨஌͠ɺεΫϦϓτͳͲͷ
    ࣮ߦ͕Մೳ

    View Slide

  59. 4FSG
    TFSGNFNCFST
    MCEFWMPDBMBMJWFSPMFMC
    EOTEFWMPDBMBMJWFSPMFEOT
    EOTEFWMPDBMBMJWFSPMFEOT
    DBUBHFOUMPH
    BHFOU&WFOUNFNCFSKPJOEOTEFWMPDBMEOTSPMFEOT
    ಉ͡άϧʔϓʹॴଐ͍ͯ͠Δϗετ
    ৽͘͠ϝϯόʔ͕௥Ճ͞Εͨ

    View Slide

  60. ࣗಈԽ4UFQ
    $IFG
    7.
    TFSGNFNCFSKPJO
    -#ϨγϐΛྲྀ͢
    -#
    7.ͷىಈͱಉ࣌ʹ

    ɹDMPVEJOJUͰTFSGΠϯετʔϧ
    Ϩγϐ͕ྲྀΕͯ-#ͷΠϯετʔϧ͕׬ྃ

    View Slide

  61. ࣗಈԽ4UFQ
    -# -#
    %/4 %/4 %/4
    WJQ
    SJQ SJQ

    View Slide

  62. ࣗಈԽ4UFQ
    $IFG
    7.
    EOTϨγϐΛྲྀ͢
    %/4
    Ϩγϐ͕ྲྀΕͯEOTͷΠϯετʔϧ͕׬ྃ
    -#
    TFSGͷNFNCFSKPJO
    %/4αʔϏεͷ7*1ʹKPJOͨ͠%/4Λ
    ɹϝϯόʔ௥Ճ
    7.ͷىಈͱಉ࣌ʹ

    ɹDMPVEJOJUͰTFSGΠϯετʔϧ
    TFSGNFNCFSKPJO

    View Slide

  63. ࣗಈԽ4UFQ
    -# -#
    %/4 %/4 %/4
    WJQ
    SJQ SJQ
    SJQ
    SJQ
    SJQ

    View Slide

  64. ࣗಈԽ4UFQ
    $IFG
    7.
    EOTϨγϐΛྲྀ͢
    %/4
    Ϩγϐ͕ྲྀΕͯEOTͷΠϯετʔϧ͕׬ྃ
    -#
    TFSGͷNFNCFSKPJO
    %/4αʔϏεͷ7*1ʹKPJOͨ͠%/4Λ
    ɹϝϯόʔ௥Ճ
    7.ͷىಈͱಉ࣌ʹ

    ɹDMPVEJOJUͰTFSGΠϯετʔϧ
    TFSGͷNFNCFSKPJO

    View Slide

  65. /BNJOPSJ
    4FSG-WTͷϥΠϒϥϦ
    αʔϏεͷϔϧενΣοΫ΋Ͱ͖Δ
    ϝϯόʔͷ௥Ճ΍࡟আΛ4MBDL௨஌
    #! /usr/bin/env ruby
    require 'rubygems'
    require 'naminori'
    Naminori::Notifier::Configure.instance.set({
    webhook_url: "https://hooks.slack.com/services/XXXXXX",
    channel: "#pyama",
    user: "#naminori"
    })
    service_options = {
    vip:"192.168.77.9"
    }
    case
    when Naminori::Serf.role?("dns")
    Naminori::Service.event("dns", "lvs", service_options)
    when Naminori::Serf.role?("lb")
    Naminori::Lb.health_check("dns", "lvs",service_options)
    end

    View Slide

  66. ·ͱΊ

    View Slide

  67. શମ·ͱΊ
    ϓϥΠϕʔτΫϥ΢υΛαʔϏε͝ͱʹς
    φϯτ؅ཧ͠ӡ༻͍ͯ͠·͢
    ߏ੒؅ཧ΍؂ࢹ͸ࣗಈԽ
    ख͕͔͔ؒΔ͜ͱɺ؅ཧ͕ඞཁͳ͜ͱ͸ࣗ
    ෼ͨͪͰίʔυॻ͍ͯղܾ
    αʔϏεΛҠߦ͢Δ͚ͩͰ͸ͳͯ͘ɺΫϥ
    ΢υΒ͍͜͠ͱ΋࢝Ίͯ·͢

    View Slide

  68. ͜͜Ͱঁੑਞ͔Β

    ࣭໰͕ࡴ౸͢Δ

    View Slide

  69. ͝ਗ਼ௌ͋Γ͕ͱ͏

    ͍͟͝·ͨ͠

    View Slide

  70. Ҡߦࣄྫ

    View Slide

  71. /ZBIͰಈ͍͍ͯΔαʔϏε
    ֤αʔϏεӶҙҾͬӽ͠த

    View Slide

  72. View Slide

  73. 4UFQ
    Ҡߦݩ
    -#
    1SPYZ
    8
    5IF*OUFSOFU

    View Slide

  74. 4UFQ
    Ҡߦݩ
    -#
    1SPYZ
    8
    -#
    1SPYZ
    8
    5IF*OUFSOFU

    View Slide

  75. 4UFQ
    Ҡߦݩ
    -#
    1SPYZ
    8
    -#
    1SPYZ
    8
    5IF*OUFSOFU

    View Slide

  76. ϜʔϜʔυϝΠϯ
    ΠϯϑϥΛ͢΂ͯ$IFGͰίʔυԽ
    յΕͨΒࣺͯͯɺ࡞Γ௚͢
    -#ͷ%JSFDU3PVUJOHߏ੒͕

    ૊Ίͳ͍ʜ

    View Slide

  77. %3ߏ੒૊Ίͳ͍໰୊
    /PWB$PNQVUF͸εςʔτϑϧʹ൑அ͠

    *OWBMJE1BDLFUΛഁغ͢Δ
    5$1XBZϋϯυγΣΠΫ͕੒ཱ͠ͳ͘ͳͬͨ
    IPTU#
    IPTU"
    TZO
    TZOBDL
    BDL
    σʔλ௨৴ͷ։࢝

    View Slide

  78. DPNQVUF
    DPNQVUF
    %3ߏ੒૊Ίͳ͍໰୊
    -#
    1SPYZ
    TZO
    TZO
    TZOBDL
    TZOBDLΛฦͨ͠ͷ͸DPNQVUF
    DMJFOU

    View Slide

  79. DPNQVUF
    DPNQVUF
    %3ߏ੒૊Ίͳ͍໰୊
    -#
    1SPYZ
    BDL
    DMJFOU

    View Slide

  80. DPNQVUF
    DPNQVUF
    /"5ߏ੒ʹͨ͠
    -#
    1SPYZ
    TZO
    TZO
    TZOBDL
    DMJFOU

    View Slide

  81. ϜʔϜʔυϝΠϯҠߦࣄྫ·ͱΊ
    Πϯϑϥ͸͢΂ͯ$IFGͰίʔυԽ
    7.ͷσʔλ͸͢΂ͯ$PNQVUF
    OPEF ෺ཧϗετ
    ͷϩʔΧϧ

    σΟεΫͷͨΊɺյΕͨΒ࡞Γͳ͓
    ͢
    -#ͷ%3ߏ੒͸૊Ίͳ͍

    View Slide