Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Hardening におけるトラブルシューティング / Troubleshooting in Hardening

Hardening におけるトラブルシューティング / Troubleshooting in Hardening

Hardening 2020 Deep Digital Dependence の登壇資料です。
https://wasforum.jp/2020/08/hardening-2020-deep-digital-dependence/

Takamura Narimichi

September 05, 2020
Tweet

More Decks by Takamura Narimichi

Other Decks in Technology

Transcript

  1. Hardening ʹ͓͚Δ
    τϥϒϧγϡʔςΟϯά
    גࣜձࣾϋʔτϏʔπ
    औక໾ ߴଜ ੒ಓ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex

    View full-size slide

  2. about me
    • ॴଐ
    • גࣜձࣾϋʔτϏʔπ औక໾
    • ޷͖ͳήʔϜ
    • ετϦʔτϑΝΠλʔV
    • Hardening
    • 2020 BO ͰάϥϯϓϦड৆
    • ΑΓৄࡉͳ৘ใ͸ͪ͜Β΁
    • h1ps:/
    /www.nari-ex.com/about/
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 2

    View full-size slide

  3. גࣜձࣾϋʔτϏʔπ
    • 2005೥૑ۀ
    • ࣄۀ಺༰
    • MSP ࣄۀ
    • Πϯϑϥӡ༻ͷΞ΢τιʔαʔ
    • ؂ࢹɺઃܭɺߏஙɺΫϥ΢υಋೖࢧ
    ԉɺίϯαϧςΟϯά
    • ։ൃࣄۀ
    • ࣾһ਺: 71໊ʢ໿8ׂҎ্͕ΤϯδχΞʣ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 3

    View full-size slide

  4. ͜ͷηογϣϯʹ͍ͭͯ
    • Hardening ࣗମͷઆ໌͸ׂѪ͠·͢
    • Hardening ͷো֐ରԠΛߦ͏্ͰॏཁͳϙΠϯτΛ͓࿩͠·͢
    • τϥϒϧγϡʔςΟϯάະܦݧͷํ޲͚ͷࢿྉͰ͢
    • ۩ମతͳରԠৄࡉʢઃఆ΍ίϚϯυͳͲʣͷ࿩͸͠·ͤΜ
    • SRE ΍ IMS ͷ஌ݟ΋౿·͑ͯൃද͠·͢
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 4

    View full-size slide

  5. Agenda
    • SRE ͱ IMS ͔Βͷֶͼ
    • τϥϒϧγϡʔςΟϯά
    • ΠϯγσϯτϚωδϝϯτ
    • Hardening BO 2020 Ͱ࣮ࡍʹ΍ͬͨ͜ͱ
    • Hardening ʹ͓͚Δো֐ରԠͷϙΠϯτ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 5

    View full-size slide

  6. τϥϒϧγϡʔςΟϯά

    View full-size slide

  7. Site Reliability Engineering
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 7

    View full-size slide

  8. SREຊʹ͓͚Δো֐ରԠʹؔ࿈͢Δষ
    • 12ষ: ޮ཰తͳτϥϒϧγϡʔςΟϯά
    • 13ষ: ۓٸରԠ
    • 14ষ: Πϯγσϯτ؅ཧ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 8

    View full-size slide

  9. ෺ࣄ͕͏·͍͘͘৔߹ͱ͍͏΋ͷ
    ͸ɺ෺ࣄ͕͓͔͘͠ͳΔ৔߹ͷதͷ
    ಛघͳྫʹա͗ͳ͍ɻ
    — John Allspaw
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 9

    View full-size slide

  10. τϥϒϧγϡʔςΟϯάͷϓ
    ϩηε
    • ໰୊ͷϨϙʔτ
    • ظ଴͞ΕΔಈ࡞ͱ࣮ࡍͷಈ࡞Λࣔ͢
    • ex. ࣗಈԽ͞Εͨ؂ࢹγεςϜʹΑΔΞϥʔτ௨஌
    • ex. ಉ྅͔ΒʮγεςϜ͕஗͘ͳ͍ͬͯΔʯͱฉ͍ͨ
    • τϦΞʔδ
    • ໰୊ͷॏཁ౓Λ൑அ্ͨ͠ͰͳʹΛ͢΂͖͔൑அ͢Δ
    • ؍࡯ɺ਍அ
    • ର৅ͷγεςϜͷ৘ใΛݩʹݪҼΛಛఆ͢Δ
    • ςετ/ରॲɺ෮چ
    • ಛఆͨ͠ݪҼ΁ͷରࡦΛߦ͍ɺ෮چͤ͞Δ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 10

    View full-size slide

  11. τϥϒϧγϡʔςΟϯάͷϓϩηεΛཧղ͢Δ͚ͩͰ͸ෆे෼
    Hardening BO 2020 Ͱ͸͜Μͳ؀ڥͰͨ͠
    • ॳݟ ͷγεςϜ܈
    • 6ʙ8໊ఔ౓ͷνʔϜ Ͱ࡞ۀΛߦ͏
    • ސ٬ͱͷίϛϡχέʔγϣϯ ͕ੜ͡Δ
    • ෺ཧతʹ෼཭ ͨ͠ΦϖϨʔγϣϯࣨ
    • ۮવى͖Δো֐Ͱ͸ͳ͘ɺഁյΛ໨తͱͨ͠߈ܸ΁ͷରԠ
    • ...etc
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 11

    View full-size slide

  12. Hardening ͷτϥϒϧγϡʔςΟϯά
    • ຖ೔৮ͬͯΔγεςϜͷখ͞ͳো֐Λࣗ෼ҰਓͰαΫοͱରԠ
    ͢ΔΑ͏ͳ΋ͷͰ͸ͳ͍
    • → ໨ͷલͷγεςϜ͚ͩͰͳ͘ɺো֐ରԠʹཱͪ޲͔͏ਓͷ໾
    ׂ΍ମ੍΋౿·͑ͯߟ͑Δ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 12

    View full-size slide

  13. ΠϯγσϯτϚωδϝϯτ

    View full-size slide

  14. “Fire is not an
    emergency to the fire
    department. It’s what
    we do.”
    Րࡂ͸ফ๷ॺʹͱͬͯۓٸࣄଶͰ͸
    ͳ͍ɻͦΕ͸զʑͷ΍Δ͜ͱͩɻ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 14

    View full-size slide

  15. Incident Management for Opera2ons
    • ফ๷΍ॏཁΠϯϑϥʹؔΘ͖ͬͯͨਓ͕ͨͪ IMS Λ IT ӡ༻ʹ
    ద༻ͤͨ͞΋ͷΛ঺հ͍ͯ͠Δ
    • IMS ͱ͸ɺ΋ͱ΋ͱ40೥Ҏ্ʹ౉Γɺશถͷফ๷ॺ͕͋ΒΏΔ
    छྨͷۓٸࣄଶʹରԠ͢ΔͨΊʹར༻͖ͯͨ͠ϑϨʔϜϫʔΫ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 15

    View full-size slide

  16. ࢀߟ: ΠϯγσϯτϥΠϑαΠΫϧ
    ΠϯγσϯτରԠͷମ੍΋౿·͑ͯରԠΛߦ͏
    1. ໰୊Λݕ஌͢Δ
    2. ໰୊͕Πϯγσϯτͳͷ͔Πϕϯτͳͷ͔൑அ͢Δ
    3. ద੾ͳΠϯγσϯτରԠνʔϜΛฤ੒͢Δ
    4. ରԠνʔϜͷฏۉ૊Έཱͯ࣌ؒʢMTTAʣΛ૊ΈཱͯΔ
    5. ΠϯγσϯτίϚϯμʔΛཱ֬͠ɺϦιʔεΛ੔ཧ͠ɺΠϯγσϯτͷ໨తΛઃఆ͢Δ
    6. IMSΛར༻ͯ͠ɺରԠΛߦ͏
    7. ద੾ͳར֐ؔ܎ऀʹ௨஌͢Δ
    8. ΠϯγσϯτΛղܾͯ͠ɺϦιʔεΛղ์͢Δ
    9. AARʢA1er Ac6on ReviewʣΛ࣮ࢪ͢Δ
    10. ඼࣭վળʢQIʣͱ඼࣭อূʢQAʣΛ࣮ࢪ͢Δ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 16

    View full-size slide

  17. ΠϯγσϯτରԠνʔϜ
    • Incident Commander (IC)
    • ΠϯγσϯτରԠͷϦʔμʔ
    • ҙࢥܾఆɺϑΝγϦςʔλʔ
    • Communica2on Officer (Comms)
    • ICΛิࠤ͠ɺؔ܎ऀʹ࿈བྷΛߦ͏ਓ
    • Situa2onal Status (Scribe)
    • Πϯγσϯτʹؔ࿈ͨ͠৘ใΛه࿥͢Δਓʢॻهʣ
    • Group Leader
    • ઐ໳ՈάϧʔϓͷϦʔμʔɻ࣮ࡍʹରԠΛऔΓ࢓੾Δਓ
    • Subject Ma>er Expert (SME)
    • ઐ໳Ո
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 17

    View full-size slide

  18. ͜͜·Ͱͷ·ͱΊ
    • τϥϒϧγϡʔςΟϯά
    • ໰୊ͷϨϙʔτ͕ى఺
    • ԾઆɾݕূΛ܁Γฦ͢͜ͱͰγεςϜΛ෮چʹಋ͘
    • ΠϯγσϯτϚωδϝϯτ
    • ΠϯγσϯτίϚϯμʔ͕ରԠͷத৺
    • ໾ׂ͸ணखલʹܾΊΔ
    • ࣮ରԠ͢Δ໾ׂʢSMEʣҎ֎΋ׂͱ͋Δ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 18

    View full-size slide

  19. Hardening 2020 BO Ͱ
    ࣮ࡍʹ΍ͬͨ͜ͱ

    View full-size slide

  20. ͜Ε·Ͱͷ஌ݟΛࣄલ४උʹద༻
    • γεςϜҎ֎ͷ୲౰΋ؚΊͯ໾ׂ෼୲Λߦ͏
    • ΠϯγσϯτίϚϯμʔΛ໌֬ʹ͢Δ
    • ҟৗݕ஌ͷ࢓૊ΈΛߟ͑Δ
    • ؂ࢹγεςϜʹΑΔݕ஌
    • ਓʹΑΔݕ஌
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 20

    View full-size slide

  21. લ೔ ·Ͱ ʹ΍ͬͨ͜ͱ
    • ໾ׂͷΞαΠϯ
    • શһͰإ߹ΘͤʢҿΈձʣ
    • τϥϒϧγϡʔςΟϯά୲౰಺Ͱͷଧͪ߹Θͤ
    • γεςϜ୲౰͸ߏ੒೺Ѳ
    • ҰਓͻͱΓ͕ॳಈ࣌ʹ΍Δ͜ͱΛܾఆ
    • αʔϏεURLҰཡͷ४උ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 21

    View full-size slide

  22. ໾ׂ෼୲ͷ༷ࢠ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 22

    View full-size slide

  23. ໾ׂ෼୲ͷ಺༰
    • ΠϯγσϯτίϚϯμʔΛࣄલʹܾఆ && ݕ஌ͱରԠͷ໾ׂΛ۠ผ
    • ۩ମతʹ͸ҎԼͷ5ͭͷ໾ׂΛׂΓ౰ͯ
    • ΠϯγσϯτίϚϯμʔʢλεΫ༏ઌ౓ɺܦࡁ৚݅ͷҙࢥܾఆͳͲʣ
    • τϥϒϧγϡʔςΟϯά1ʢݕ஌ɺ੾Γ෼͚ʣ
    • τϥϒϧγϡʔςΟϯά2ʢରࡦ࣮ࢪʣ
    • ಺෦΁ͷϨϙʔλʔ݉ব֎
    • ൢച؅ཧʢαʔϏεʹΑΔച্ͷ؅ཧͳͲʣ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 23

    View full-size slide

  24. ։࢝௚ޙʹ΍ͬͨ͜ͱ
    • ؂ࢹπʔϧʢZabbixʣ ʹΑΔ؂ࢹઃఆ
    • ར༻͍ͯ͠ͳ͍ϛυϧ΢ΣΞͷఀࢭɾ࡟আ
    • Rainloop, phpmyadmin, phppgmyadmin ͳͲ
    • ֤ॴͷύεϫʔυมߋ
    • ֤σʔλͷόοΫΞοϓ
    • ෆཁͳωοτϫʔΫΞΫηεͷःஅ
    • ex. WAN ͔Β DMZ ΁ͷ DB ϙʔτ΁ͷΞΫηεःஅ
    • IPS ͷಋೖʢCisco ༷ఏڙʣ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 24

    View full-size slide

  25. ౰೔ͷτϥϒϧγϡʔςΟϯά
    • Өڹൣғͷେ͖͍ো֐Λ༏ઌతʹରॲ
    • ඞཁʹԠͯ͡ো֐ରԠؒͰϖΞΦϖ
    • ணखͱ෮چͷ࿈བྷ͸͜·Ίʹ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 25

    View full-size slide

  26. ࢀߟ: ։࢝࣌ͷෆඋɾෆ۩߹ʹΑΔো֐
    • ࠾༻αΠτ: DB ઀ଓઌͷ࠶ઃఆ
    • νϟοτ: Docker ίϯςφΛਖ਼͘͠ىಈ
    • Shop3: ࣄલ഑෍ͷυΩϡϝϯτͷURL͕͍ؒͬͯΔ
    • Shop3: POST ϦΫΤετ࣌ͷϦΫΤεταζ্ݶ͕10KͰ͋ͬͨͨΊ͕Ҿ্͖͛
    • m-pay: υϝΠϯ໊Λਖ਼͍͠΋ͷʹSQLʹ͖ͯ׵͑
    • m-pay: ϓϥάΠϯϑΝΠϧͷΞΫηεݖݶमਖ਼
    • Nginx: Nginx + PHP-FPM +fastcgi_split_path ͷ૊Έ߹ΘͤʹΑΔऑੑͷमਖ਼
    • EC-CUBE: ΠϯετʔϧεΫϦϓτͷୀආ
    • EC-CUBE: େྲྀߦͨ͠ط஌ͷ੬ऑੑ΁ͷରࡦ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 26

    View full-size slide

  27. ࢀߟ: ։࢝ޙͷ߈ܸʹΑΔো֐
    • Shop: iptables ͷ INPUT νΣΠϯʹ443, 80 ϙʔτ΁ͷ઀ଓΛ DROP ͢Δϧʔϧ௥Ճ͞Ε͍ͯ
    Δ
    • શମ: αʔό಺Ͱ໊લղܾ͕Ͱ͖ͳ͍
    • srv4,6: ΫϨδοτΧʔυ৘ใΛҾͬ͜ൈ͘JS ϑΝΠϧ͕ஔ͔Ε͍ͯΔ
    • શମ: ϛυϧ΢ΣΞ͕஌Β͵ؒʹམ͍ͪͯΔ
    • Shop: PHP-FPM ͕ਖ਼ৗʹಈ࡞͠ͳ͍
    • srv4,6: 25൪ϙʔτΛར༻ͯ͠֎෦΁େྔσʔλ͕ૹ৴͞Ε͍ͯΔʢΫϨΧ৘ใྲྀग़ʁʣ
    • srv7: όφʔͷը૾͕ϋοΩϯάޙͷը૾ʹͳ͍ͬͯΔ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 27

    View full-size slide

  28. ো֐ใࠂνϟϯωϧͷ༷ࢠ

    View full-size slide

  29. Hardening ʹ͓͚Δ
    ো֐ରԠͷϙΠϯτ

    View full-size slide

  30. Hardening ʹ͓͚Δো֐ରԠͷϙΠϯτ
    • ։࢝લͱ։࢝௚ޙͷ४උ͕େ੾
    • ಛʹҎԼͷ2఺͕ॏཁ
    • ໾ׂ෼୲
    • ҟৗݕ஌
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 30

    View full-size slide

  31. ໾ׂ෼୲
    • ΠϯγσϯτίϚϯμʔͷઃஔ
    • ݕ஌ͱରԠͷ෼཭
    • ࣗνʔϜͰ͸ɺγεςϜʹ͸೔ʑ৮Ε͍ͯΔ͚Ͳো֐ରԠͷ
    ܦݧ͸͋·Γͳ͍ํ͕ͨͪݕ஌໾Ͱͨ͠
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 31

    View full-size slide

  32. ҟৗݕ஌
    • ඞͣ؂ࢹγεςϜͱਓͷ໨ͷ྆ํͰߦ͏
    • ಥ؏ͰೖΕͨ؂ࢹઃఆ͸࿙Ε͕ଟ͍ͷͰɺਓ͕αʔϏε΍ε
    ίΞΛఆظతʹ֬ೝ͢Δͷ͸ॏཁ
    • ҟৗͷ࿈བྷ͸Ͱ͖Δ͚ͩه࿥͕࢒ΔܗࣜͰߦ͏
    • ޱ಄ͩͱ৘ใ͕شൃͯ͠๨Εͯ͠·͏ͨΊ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 32

    View full-size slide

  33. ·ͱΊ
    • SRE ΍ IMS ΛݩʹτϥϒϧγϡʔςΟϯά΍ΠϯγσϯτϚω
    δϝϯτͷ஌ݟΛ঺հ͠·ͨ͠
    • Hardening Ͱ࣮ࡍʹߦͬͨ͜ͱΛ঺հ͠ɺϙΠϯτΛ·ͱΊ·
    ͨ͠
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 33

    View full-size slide