Hardening におけるトラブルシューティング / Troubleshooting in Hardening

Hardening 2020 Deep Digital Dependence の登壇資料です。
https://wasforum.jp/2020/08/hardening-2020-deep-digital-dependence/

Transcript

1. Hardening ʹ͓͚Δ
   τϥϒϧγϡʔςΟϯά
   גࣜձࣾϋʔτϏʔπ
   औక໾ ߴଜ ੒ಓ
   2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex
   

   View Slide

2. about me
   • ॴଐ
   • גࣜձࣾϋʔτϏʔπ औక໾
   • ޷͖ͳήʔϜ
   • ετϦʔτϑΝΠλʔV
   • Hardening
   • 2020 BO ͰάϥϯϓϦड৆
   • ΑΓৄࡉͳ৘ใ͸ͪ͜Β΁
   • h1ps:/
   /www.nari-ex.com/about/
   2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 2
   

   View Slide

3. גࣜձࣾϋʔτϏʔπ
   • 2005೥૑ۀ
   • ࣄۀ಺༰
   • MSP ࣄۀ
   • Πϯϑϥӡ༻ͷΞ΢τιʔαʔ
   • ؂ࢹɺઃܭɺߏஙɺΫϥ΢υಋೖࢧ
   ԉɺίϯαϧςΟϯά
   • ։ൃࣄۀ
   • ࣾһ਺: 71໊ʢ໿8ׂҎ্͕ΤϯδχΞʣ
   2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 3
   

   View Slide

4. ͜ͷηογϣϯʹ͍ͭͯ
   • Hardening ࣗମͷઆ໌͸ׂѪ͠·͢
   • Hardening ͷো֐ରԠΛߦ͏্ͰॏཁͳϙΠϯτΛ͓࿩͠·͢
   • τϥϒϧγϡʔςΟϯάະܦݧͷํ޲͚ͷࢿྉͰ͢
   • ۩ମతͳରԠৄࡉʢઃఆ΍ίϚϯυͳͲʣͷ࿩͸͠·ͤΜ
   • SRE ΍ IMS ͷ஌ݟ΋౿·͑ͯൃද͠·͢
   2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 4
   

   View Slide

5. Agenda
   • SRE ͱ IMS ͔Βͷֶͼ
   • τϥϒϧγϡʔςΟϯά
   • ΠϯγσϯτϚωδϝϯτ
   • Hardening BO 2020 Ͱ࣮ࡍʹ΍ͬͨ͜ͱ
   • Hardening ʹ͓͚Δো֐ରԠͷϙΠϯτ
   2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 5
   

   View Slide

6. τϥϒϧγϡʔςΟϯά
   

   View Slide

7. Site Reliability Engineering
   2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 7
   

   View Slide

8. SREຊʹ͓͚Δো֐ରԠʹؔ࿈͢Δষ
   • 12ষ: ޮ཰తͳτϥϒϧγϡʔςΟϯά
   • 13ষ: ۓٸରԠ
   • 14ষ: Πϯγσϯτ؅ཧ
   2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 8
   

   View Slide

9. ෺ࣄ͕͏·͍͘͘৔߹ͱ͍͏΋ͷ
   ͸ɺ෺ࣄ͕͓͔͘͠ͳΔ৔߹ͷதͷ
   ಛघͳྫʹա͗ͳ͍ɻ
   — John Allspaw
   2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 9
   

   View Slide

10. τϥϒϧγϡʔςΟϯάͷϓ
    ϩηε
    • ໰୊ͷϨϙʔτ
    • ظ଴͞ΕΔಈ࡞ͱ࣮ࡍͷಈ࡞Λࣔ͢
    • ex. ࣗಈԽ͞Εͨ؂ࢹγεςϜʹΑΔΞϥʔτ௨஌
    • ex. ಉ྅͔ΒʮγεςϜ͕஗͘ͳ͍ͬͯΔʯͱฉ͍ͨ
    • τϦΞʔδ
    • ໰୊ͷॏཁ౓Λ൑அ্ͨ͠ͰͳʹΛ͢΂͖͔൑அ͢Δ
    • ؍࡯ɺ਍அ
    • ର৅ͷγεςϜͷ৘ใΛݩʹݪҼΛಛఆ͢Δ
    • ςετ/ରॲɺ෮چ
    • ಛఆͨ͠ݪҼ΁ͷରࡦΛߦ͍ɺ෮چͤ͞Δ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 10
    

    View Slide

11. τϥϒϧγϡʔςΟϯάͷϓϩηεΛཧղ͢Δ͚ͩͰ͸ෆे෼
    Hardening BO 2020 Ͱ͸͜Μͳ؀ڥͰͨ͠
    • ॳݟ ͷγεςϜ܈
    • 6ʙ8໊ఔ౓ͷνʔϜ Ͱ࡞ۀΛߦ͏
    • ސ٬ͱͷίϛϡχέʔγϣϯ ͕ੜ͡Δ
    • ෺ཧతʹ෼཭ ͨ͠ΦϖϨʔγϣϯࣨ
    • ۮવى͖Δো֐Ͱ͸ͳ͘ɺഁյΛ໨తͱͨ͠߈ܸ΁ͷରԠ
    • ...etc
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 11
    

    View Slide

12. Hardening ͷτϥϒϧγϡʔςΟϯά
    • ຖ೔৮ͬͯΔγεςϜͷখ͞ͳো֐Λࣗ෼ҰਓͰαΫοͱରԠ
    ͢ΔΑ͏ͳ΋ͷͰ͸ͳ͍
    • → ໨ͷલͷγεςϜ͚ͩͰͳ͘ɺো֐ରԠʹཱͪ޲͔͏ਓͷ໾
    ׂ΍ମ੍΋౿·͑ͯߟ͑Δ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 12
    

    View Slide

13. ΠϯγσϯτϚωδϝϯτ
    

    View Slide

14. “Fire is not an
    emergency to the fire
    department. It’s what
    we do.”
    Րࡂ͸ফ๷ॺʹͱͬͯۓٸࣄଶͰ͸
    ͳ͍ɻͦΕ͸զʑͷ΍Δ͜ͱͩɻ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 14
    

    View Slide

15. Incident Management for Opera2ons
    • ফ๷΍ॏཁΠϯϑϥʹؔΘ͖ͬͯͨਓ͕ͨͪ IMS Λ IT ӡ༻ʹ
    ద༻ͤͨ͞΋ͷΛ঺հ͍ͯ͠Δ
    • IMS ͱ͸ɺ΋ͱ΋ͱ40೥Ҏ্ʹ౉Γɺશถͷফ๷ॺ͕͋ΒΏΔ
    छྨͷۓٸࣄଶʹରԠ͢ΔͨΊʹར༻͖ͯͨ͠ϑϨʔϜϫʔΫ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 15
    

    View Slide

16. ࢀߟ: ΠϯγσϯτϥΠϑαΠΫϧ
    ΠϯγσϯτରԠͷମ੍΋౿·͑ͯରԠΛߦ͏
    1. ໰୊Λݕ஌͢Δ
    2. ໰୊͕Πϯγσϯτͳͷ͔Πϕϯτͳͷ͔൑அ͢Δ
    3. ద੾ͳΠϯγσϯτରԠνʔϜΛฤ੒͢Δ
    4. ରԠνʔϜͷฏۉ૊Έཱͯ࣌ؒʢMTTAʣΛ૊ΈཱͯΔ
    5. ΠϯγσϯτίϚϯμʔΛཱ֬͠ɺϦιʔεΛ੔ཧ͠ɺΠϯγσϯτͷ໨తΛઃఆ͢Δ
    6. IMSΛར༻ͯ͠ɺରԠΛߦ͏
    7. ద੾ͳར֐ؔ܎ऀʹ௨஌͢Δ
    8. ΠϯγσϯτΛղܾͯ͠ɺϦιʔεΛղ์͢Δ
    9. AARʢA1er Ac6on ReviewʣΛ࣮ࢪ͢Δ
    10. ඼࣭վળʢQIʣͱ඼࣭อূʢQAʣΛ࣮ࢪ͢Δ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 16
    

    View Slide

17. ΠϯγσϯτରԠνʔϜ
    • Incident Commander (IC)
    • ΠϯγσϯτରԠͷϦʔμʔ
    • ҙࢥܾఆɺϑΝγϦςʔλʔ
    • Communica2on Officer (Comms)
    • ICΛิࠤ͠ɺؔ܎ऀʹ࿈བྷΛߦ͏ਓ
    • Situa2onal Status (Scribe)
    • Πϯγσϯτʹؔ࿈ͨ͠৘ใΛه࿥͢Δਓʢॻهʣ
    • Group Leader
    • ઐ໳ՈάϧʔϓͷϦʔμʔɻ࣮ࡍʹରԠΛऔΓ࢓੾Δਓ
    • Subject Ma>er Expert (SME)
    • ઐ໳Ո
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 17
    

    View Slide

18. ͜͜·Ͱͷ·ͱΊ
    • τϥϒϧγϡʔςΟϯά
    • ໰୊ͷϨϙʔτ͕ى఺
    • ԾઆɾݕূΛ܁Γฦ͢͜ͱͰγεςϜΛ෮چʹಋ͘
    • ΠϯγσϯτϚωδϝϯτ
    • ΠϯγσϯτίϚϯμʔ͕ରԠͷத৺
    • ໾ׂ͸ணखલʹܾΊΔ
    • ࣮ରԠ͢Δ໾ׂʢSMEʣҎ֎΋ׂͱ͋Δ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 18
    

    View Slide

19. Hardening 2020 BO Ͱ
    ࣮ࡍʹ΍ͬͨ͜ͱ
    

    View Slide

20. ͜Ε·Ͱͷ஌ݟΛࣄલ४උʹద༻
    • γεςϜҎ֎ͷ୲౰΋ؚΊͯ໾ׂ෼୲Λߦ͏
    • ΠϯγσϯτίϚϯμʔΛ໌֬ʹ͢Δ
    • ҟৗݕ஌ͷ࢓૊ΈΛߟ͑Δ
    • ؂ࢹγεςϜʹΑΔݕ஌
    • ਓʹΑΔݕ஌
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 20
    

    View Slide

21. લ೔ ·Ͱ ʹ΍ͬͨ͜ͱ
    • ໾ׂͷΞαΠϯ
    • શһͰإ߹ΘͤʢҿΈձʣ
    • τϥϒϧγϡʔςΟϯά୲౰಺Ͱͷଧͪ߹Θͤ
    • γεςϜ୲౰͸ߏ੒೺Ѳ
    • ҰਓͻͱΓ͕ॳಈ࣌ʹ΍Δ͜ͱΛܾఆ
    • αʔϏεURLҰཡͷ४උ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 21
    

    View Slide

22. ໾ׂ෼୲ͷ༷ࢠ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 22
    

    View Slide

23. ໾ׂ෼୲ͷ಺༰
    • ΠϯγσϯτίϚϯμʔΛࣄલʹܾఆ && ݕ஌ͱରԠͷ໾ׂΛ۠ผ
    • ۩ମతʹ͸ҎԼͷ5ͭͷ໾ׂΛׂΓ౰ͯ
    • ΠϯγσϯτίϚϯμʔʢλεΫ༏ઌ౓ɺܦࡁ৚݅ͷҙࢥܾఆͳͲʣ
    • τϥϒϧγϡʔςΟϯά1ʢݕ஌ɺ੾Γ෼͚ʣ
    • τϥϒϧγϡʔςΟϯά2ʢରࡦ࣮ࢪʣ
    • ಺෦΁ͷϨϙʔλʔ݉ব֎
    • ൢച؅ཧʢαʔϏεʹΑΔച্ͷ؅ཧͳͲʣ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 23
    

    View Slide

24. ։࢝௚ޙʹ΍ͬͨ͜ͱ
    • ؂ࢹπʔϧʢZabbixʣ ʹΑΔ؂ࢹઃఆ
    • ར༻͍ͯ͠ͳ͍ϛυϧ΢ΣΞͷఀࢭɾ࡟আ
    • Rainloop, phpmyadmin, phppgmyadmin ͳͲ
    • ֤ॴͷύεϫʔυมߋ
    • ֤σʔλͷόοΫΞοϓ
    • ෆཁͳωοτϫʔΫΞΫηεͷःஅ
    • ex. WAN ͔Β DMZ ΁ͷ DB ϙʔτ΁ͷΞΫηεःஅ
    • IPS ͷಋೖʢCisco ༷ఏڙʣ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 24
    

    View Slide

25. ౰೔ͷτϥϒϧγϡʔςΟϯά
    • Өڹൣғͷେ͖͍ো֐Λ༏ઌతʹରॲ
    • ඞཁʹԠͯ͡ো֐ରԠؒͰϖΞΦϖ
    • ணखͱ෮چͷ࿈བྷ͸͜·Ίʹ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 25
    

    View Slide

26. ࢀߟ: ։࢝࣌ͷෆඋɾෆ۩߹ʹΑΔো֐
    • ࠾༻αΠτ: DB ઀ଓઌͷ࠶ઃఆ
    • νϟοτ: Docker ίϯςφΛਖ਼͘͠ىಈ
    • Shop3: ࣄલ഑෍ͷυΩϡϝϯτͷURL͕͍ؒͬͯΔ
    • Shop3: POST ϦΫΤετ࣌ͷϦΫΤεταζ্ݶ͕10KͰ͋ͬͨͨΊ͕Ҿ্͖͛
    • m-pay: υϝΠϯ໊Λਖ਼͍͠΋ͷʹSQLʹ͖ͯ׵͑
    • m-pay: ϓϥάΠϯϑΝΠϧͷΞΫηεݖݶमਖ਼
    • Nginx: Nginx + PHP-FPM +fastcgi_split_path ͷ૊Έ߹ΘͤʹΑΔऑੑͷमਖ਼
    • EC-CUBE: ΠϯετʔϧεΫϦϓτͷୀආ
    • EC-CUBE: େྲྀߦͨ͠ط஌ͷ੬ऑੑ΁ͷରࡦ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 26
    

    View Slide

27. ࢀߟ: ։࢝ޙͷ߈ܸʹΑΔো֐
    • Shop: iptables ͷ INPUT νΣΠϯʹ443, 80 ϙʔτ΁ͷ઀ଓΛ DROP ͢Δϧʔϧ௥Ճ͞Ε͍ͯ
    Δ
    • શମ: αʔό಺Ͱ໊લղܾ͕Ͱ͖ͳ͍
    • srv4,6: ΫϨδοτΧʔυ৘ใΛҾͬ͜ൈ͘JS ϑΝΠϧ͕ஔ͔Ε͍ͯΔ
    • શମ: ϛυϧ΢ΣΞ͕஌Β͵ؒʹམ͍ͪͯΔ
    • Shop: PHP-FPM ͕ਖ਼ৗʹಈ࡞͠ͳ͍
    • srv4,6: 25൪ϙʔτΛར༻ͯ͠֎෦΁େྔσʔλ͕ૹ৴͞Ε͍ͯΔʢΫϨΧ৘ใྲྀग़ʁʣ
    • srv7: όφʔͷը૾͕ϋοΩϯάޙͷը૾ʹͳ͍ͬͯΔ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 27
    

    View Slide

28. ো֐ใࠂνϟϯωϧͷ༷ࢠ
    

    View Slide

29. Hardening ʹ͓͚Δ
    ো֐ରԠͷϙΠϯτ
    

    View Slide

30. Hardening ʹ͓͚Δো֐ରԠͷϙΠϯτ
    • ։࢝લͱ։࢝௚ޙͷ४උ͕େ੾
    • ಛʹҎԼͷ2఺͕ॏཁ
    • ໾ׂ෼୲
    • ҟৗݕ஌
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 30
    

    View Slide

31. ໾ׂ෼୲
    • ΠϯγσϯτίϚϯμʔͷઃஔ
    • ݕ஌ͱରԠͷ෼཭
    • ࣗνʔϜͰ͸ɺγεςϜʹ͸೔ʑ৮Ε͍ͯΔ͚Ͳো֐ରԠͷ
    ܦݧ͸͋·Γͳ͍ํ͕ͨͪݕ஌໾Ͱͨ͠
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 31
    

    View Slide

32. ҟৗݕ஌
    • ඞͣ؂ࢹγεςϜͱਓͷ໨ͷ྆ํͰߦ͏
    • ಥ؏ͰೖΕͨ؂ࢹઃఆ͸࿙Ε͕ଟ͍ͷͰɺਓ͕αʔϏε΍ε
    ίΞΛఆظతʹ֬ೝ͢Δͷ͸ॏཁ
    • ҟৗͷ࿈བྷ͸Ͱ͖Δ͚ͩه࿥͕࢒ΔܗࣜͰߦ͏
    • ޱ಄ͩͱ৘ใ͕شൃͯ͠๨Εͯ͠·͏ͨΊ
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 32
    

    View Slide

33. ·ͱΊ
    • SRE ΍ IMS ΛݩʹτϥϒϧγϡʔςΟϯά΍ΠϯγσϯτϚω
    δϝϯτͷ஌ݟΛ঺հ͠·ͨ͠
    • Hardening Ͱ࣮ࡍʹߦͬͨ͜ͱΛ঺հ͠ɺϙΠϯτΛ·ͱΊ·
    ͨ͠
    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 33
    

    View Slide