Hardening におけるトラブルシューティング / Troubleshooting in Hardening

Hardening におけるトラブルシューティング / Troubleshooting in Hardening

Hardening 2020 Deep Digital Dependence の登壇資料です。
https://wasforum.jp/2020/08/hardening-2020-deep-digital-dependence/

C236ca44c4fc873f80f97b5cf8a775df?s=128

Takamura Narimichi

September 05, 2020
Tweet

Transcript

  1. Hardening ʹ͓͚Δ τϥϒϧγϡʔςΟϯά גࣜձࣾϋʔτϏʔπ औక໾ ߴଜ ੒ಓ 2020/09/04-05 Hardening 2020

    Deep Digital Dependence | @nari_ex
  2. about me • ॴଐ • גࣜձࣾϋʔτϏʔπ औక໾ • ޷͖ͳήʔϜ •

    ετϦʔτϑΝΠλʔV • Hardening • 2020 BO ͰάϥϯϓϦड৆ • ΑΓৄࡉͳ৘ใ͸ͪ͜Β΁ • h1ps:/ /www.nari64.com/about/ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 2
  3. גࣜձࣾϋʔτϏʔπ • 2005೥૑ۀ • ࣄۀ಺༰ • MSP ࣄۀ • Πϯϑϥӡ༻ͷΞ΢τιʔαʔ

    • ؂ࢹɺઃܭɺߏஙɺΫϥ΢υಋೖࢧ ԉɺίϯαϧςΟϯά • ։ൃࣄۀ • ࣾһ਺: 71໊ʢ໿8ׂҎ্͕ΤϯδχΞʣ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 3
  4. ͜ͷηογϣϯʹ͍ͭͯ • Hardening ࣗମͷઆ໌͸ׂѪ͠·͢ • Hardening ͷো֐ରԠΛߦ͏্ͰॏཁͳϙΠϯτΛ͓࿩͠·͢ • τϥϒϧγϡʔςΟϯάະܦݧͷํ޲͚ͷࢿྉͰ͢ •

    ۩ମతͳରԠৄࡉʢઃఆ΍ίϚϯυͳͲʣͷ࿩͸͠·ͤΜ • SRE ΍ IMS ͷ஌ݟ΋౿·͑ͯൃද͠·͢ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 4
  5. Agenda • SRE ͱ IMS ͔Βͷֶͼ • τϥϒϧγϡʔςΟϯά • ΠϯγσϯτϚωδϝϯτ

    • Hardening BO 2020 Ͱ࣮ࡍʹ΍ͬͨ͜ͱ • Hardening ʹ͓͚Δো֐ରԠͷϙΠϯτ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 5
  6. τϥϒϧγϡʔςΟϯά

  7. Site Reliability Engineering 2020/09/04-05 Hardening 2020 Deep Digital Dependence |

    @nari_ex 7
  8. SREຊʹ͓͚Δো֐ରԠʹؔ࿈͢Δষ • 12ষ: ޮ཰తͳτϥϒϧγϡʔςΟϯά • 13ষ: ۓٸରԠ • 14ষ: Πϯγσϯτ؅ཧ

    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 8
  9. ෺ࣄ͕͏·͍͘͘৔߹ͱ͍͏΋ͷ ͸ɺ෺ࣄ͕͓͔͘͠ͳΔ৔߹ͷதͷ ಛघͳྫʹա͗ͳ͍ɻ — John Allspaw 2020/09/04-05 Hardening 2020 Deep

    Digital Dependence | @nari_ex 9
  10. τϥϒϧγϡʔςΟϯάͷϓ ϩηε • ໰୊ͷϨϙʔτ • ظ଴͞ΕΔಈ࡞ͱ࣮ࡍͷಈ࡞Λࣔ͢ • ex. ࣗಈԽ͞Εͨ؂ࢹγεςϜʹΑΔΞϥʔτ௨஌ •

    ex. ಉ྅͔ΒʮγεςϜ͕஗͘ͳ͍ͬͯΔʯͱฉ͍ͨ • τϦΞʔδ • ໰୊ͷॏཁ౓Λ൑அ্ͨ͠ͰͳʹΛ͢΂͖͔൑அ͢Δ • ؍࡯ɺ਍அ • ର৅ͷγεςϜͷ৘ใΛݩʹݪҼΛಛఆ͢Δ • ςετ/ରॲɺ෮چ • ಛఆͨ͠ݪҼ΁ͷରࡦΛߦ͍ɺ෮چͤ͞Δ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 10
  11. τϥϒϧγϡʔςΟϯάͷϓϩηεΛཧղ͢Δ͚ͩͰ͸ෆे෼ Hardening BO 2020 Ͱ͸͜Μͳ؀ڥͰͨ͠ • ॳݟ ͷγεςϜ܈ • 6ʙ8໊ఔ౓ͷνʔϜ

    Ͱ࡞ۀΛߦ͏ • ސ٬ͱͷίϛϡχέʔγϣϯ ͕ੜ͡Δ • ෺ཧతʹ෼཭ ͨ͠ΦϖϨʔγϣϯࣨ • ۮવى͖Δো֐Ͱ͸ͳ͘ɺഁյΛ໨తͱͨ͠߈ܸ΁ͷରԠ • ...etc 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 11
  12. Hardening ͷτϥϒϧγϡʔςΟϯά • ຖ೔৮ͬͯΔγεςϜͷখ͞ͳো֐Λࣗ෼ҰਓͰαΫοͱରԠ ͢ΔΑ͏ͳ΋ͷͰ͸ͳ͍ • → ໨ͷલͷγεςϜ͚ͩͰͳ͘ɺো֐ରԠʹཱͪ޲͔͏ਓͷ໾ ׂ΍ମ੍΋౿·͑ͯߟ͑Δ 2020/09/04-05

    Hardening 2020 Deep Digital Dependence | @nari_ex 12
  13. ΠϯγσϯτϚωδϝϯτ

  14. “Fire is not an emergency to the fire department. It’s

    what we do.” Րࡂ͸ফ๷ॺʹͱͬͯۓٸࣄଶͰ͸ ͳ͍ɻͦΕ͸զʑͷ΍Δ͜ͱͩɻ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 14
  15. Incident Management for Opera2ons • ফ๷΍ॏཁΠϯϑϥʹؔΘ͖ͬͯͨਓ͕ͨͪ IMS Λ IT ӡ༻ʹ

    ద༻ͤͨ͞΋ͷΛ঺հ͍ͯ͠Δ • IMS ͱ͸ɺ΋ͱ΋ͱ40೥Ҏ্ʹ౉Γɺશถͷফ๷ॺ͕͋ΒΏΔ छྨͷۓٸࣄଶʹରԠ͢ΔͨΊʹར༻͖ͯͨ͠ϑϨʔϜϫʔΫ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 15
  16. ࢀߟ: ΠϯγσϯτϥΠϑαΠΫϧ ΠϯγσϯτରԠͷମ੍΋౿·͑ͯରԠΛߦ͏ 1. ໰୊Λݕ஌͢Δ 2. ໰୊͕Πϯγσϯτͳͷ͔Πϕϯτͳͷ͔൑அ͢Δ 3. ద੾ͳΠϯγσϯτରԠνʔϜΛฤ੒͢Δ 4.

    ରԠνʔϜͷฏۉ૊Έཱͯ࣌ؒʢMTTAʣΛ૊ΈཱͯΔ 5. ΠϯγσϯτίϚϯμʔΛཱ֬͠ɺϦιʔεΛ੔ཧ͠ɺΠϯγσϯτͷ໨తΛઃఆ͢Δ 6. IMSΛར༻ͯ͠ɺରԠΛߦ͏ 7. ద੾ͳར֐ؔ܎ऀʹ௨஌͢Δ 8. ΠϯγσϯτΛղܾͯ͠ɺϦιʔεΛղ์͢Δ 9. AARʢA1er Ac6on ReviewʣΛ࣮ࢪ͢Δ 10. ඼࣭վળʢQIʣͱ඼࣭อূʢQAʣΛ࣮ࢪ͢Δ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 16
  17. ΠϯγσϯτରԠνʔϜ • Incident Commander (IC) • ΠϯγσϯτରԠͷϦʔμʔ • ҙࢥܾఆɺϑΝγϦςʔλʔ •

    Communica2on Officer (Comms) • ICΛิࠤ͠ɺؔ܎ऀʹ࿈བྷΛߦ͏ਓ • Situa2onal Status (Scribe) • Πϯγσϯτʹؔ࿈ͨ͠৘ใΛه࿥͢Δਓʢॻهʣ • Group Leader • ઐ໳ՈάϧʔϓͷϦʔμʔɻ࣮ࡍʹରԠΛऔΓ࢓੾Δਓ • Subject Ma>er Expert (SME) • ઐ໳Ո 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 17
  18. ͜͜·Ͱͷ·ͱΊ • τϥϒϧγϡʔςΟϯά • ໰୊ͷϨϙʔτ͕ى఺ • ԾઆɾݕূΛ܁Γฦ͢͜ͱͰγεςϜΛ෮چʹಋ͘ • ΠϯγσϯτϚωδϝϯτ •

    ΠϯγσϯτίϚϯμʔ͕ରԠͷத৺ • ໾ׂ͸ணखલʹܾΊΔ • ࣮ରԠ͢Δ໾ׂʢSMEʣҎ֎΋ׂͱ͋Δ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 18
  19. Hardening 2020 BO Ͱ ࣮ࡍʹ΍ͬͨ͜ͱ

  20. ͜Ε·Ͱͷ஌ݟΛࣄલ४උʹద༻ • γεςϜҎ֎ͷ୲౰΋ؚΊͯ໾ׂ෼୲Λߦ͏ • ΠϯγσϯτίϚϯμʔΛ໌֬ʹ͢Δ • ҟৗݕ஌ͷ࢓૊ΈΛߟ͑Δ • ؂ࢹγεςϜʹΑΔݕ஌ •

    ਓʹΑΔݕ஌ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 20
  21. લ೔ ·Ͱ ʹ΍ͬͨ͜ͱ • ໾ׂͷΞαΠϯ • શһͰإ߹ΘͤʢҿΈձʣ • τϥϒϧγϡʔςΟϯά୲౰಺Ͱͷଧͪ߹Θͤ •

    γεςϜ୲౰͸ߏ੒೺Ѳ • ҰਓͻͱΓ͕ॳಈ࣌ʹ΍Δ͜ͱΛܾఆ • αʔϏεURLҰཡͷ४උ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 21
  22. ໾ׂ෼୲ͷ༷ࢠ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 22

  23. ໾ׂ෼୲ͷ಺༰ • ΠϯγσϯτίϚϯμʔΛࣄલʹܾఆ && ݕ஌ͱରԠͷ໾ׂΛ۠ผ • ۩ମతʹ͸ҎԼͷ5ͭͷ໾ׂΛׂΓ౰ͯ • ΠϯγσϯτίϚϯμʔʢλεΫ༏ઌ౓ɺܦࡁ৚݅ͷҙࢥܾఆͳͲʣ •

    τϥϒϧγϡʔςΟϯά1ʢݕ஌ɺ੾Γ෼͚ʣ • τϥϒϧγϡʔςΟϯά2ʢରࡦ࣮ࢪʣ • ಺෦΁ͷϨϙʔλʔ݉ব֎ • ൢച؅ཧʢαʔϏεʹΑΔച্ͷ؅ཧͳͲʣ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 23
  24. ։࢝௚ޙʹ΍ͬͨ͜ͱ • ؂ࢹπʔϧʢZabbixʣ ʹΑΔ؂ࢹઃఆ • ར༻͍ͯ͠ͳ͍ϛυϧ΢ΣΞͷఀࢭɾ࡟আ • Rainloop, phpmyadmin, phppgmyadmin

    ͳͲ • ֤ॴͷύεϫʔυมߋ • ֤σʔλͷόοΫΞοϓ • ෆཁͳωοτϫʔΫΞΫηεͷःஅ • ex. WAN ͔Β DMZ ΁ͷ DB ϙʔτ΁ͷΞΫηεःஅ • IPS ͷಋೖʢCisco ༷ఏڙʣ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 24
  25. ౰೔ͷτϥϒϧγϡʔςΟϯά • Өڹൣғͷେ͖͍ো֐Λ༏ઌతʹରॲ • ඞཁʹԠͯ͡ো֐ରԠؒͰϖΞΦϖ • ணखͱ෮چͷ࿈བྷ͸͜·Ίʹ 2020/09/04-05 Hardening 2020

    Deep Digital Dependence | @nari_ex 25
  26. ։࢝࣌ͷෆඋɾෆ۩߹ʹΑΔো֐ • ࠾༻αΠτ: DB ઀ଓઌͷ࠶ઃఆ • νϟοτ: Docker ίϯςφΛਖ਼͘͠ىಈ •

    Shop3: ࣄલ഑෍ͷυΩϡϝϯτͷURL͕͍ؒͬͯΔ • Shop3: POST ϦΫΤετ࣌ͷϦΫΤεταζ্ݶ͕10KͰ͋ͬͨͨΊ͕Ҿ্͖͛ • m-pay: υϝΠϯ໊Λਖ਼͍͠΋ͷʹSQLʹ͖ͯ׵͑ • m-pay: ϓϥάΠϯϑΝΠϧͷΞΫηεݖݶमਖ਼ • Nginx: Nginx + PHP-FPM +fastcgi_split_path ͷ૊Έ߹ΘͤʹΑΔऑੑͷमਖ਼ • EC-CUBE: ΠϯετʔϧεΫϦϓτͷୀආ • EC-CUBE: େྲྀߦͨ͠ط஌ͷ੬ऑੑ΁ͷରࡦ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 26
  27. ։࢝ޙͷ߈ܸʹΑΔো֐ • Shop: iptables ͷ INPUT νΣΠϯʹ443, 80 ϙʔτ΁ͷ઀ଓΛ DROP

    ͢Δϧʔϧ௥Ճ͞Ε͍ͯ Δ • શମ: αʔό಺Ͱ໊લղܾ͕Ͱ͖ͳ͍ • srv4,6: ΫϨδοτΧʔυ৘ใΛҾͬ͜ൈ͘JS ϑΝΠϧ͕ஔ͔Ε͍ͯΔ • શମ: ϛυϧ΢ΣΞ͕஌Β͵ؒʹམ͍ͪͯΔ • Shop: PHP-FPM ͕ਖ਼ৗʹಈ࡞͠ͳ͍ • srv4,6: 25൪ϙʔτΛར༻ͯ͠֎෦΁େྔσʔλ͕ૹ৴͞Ε͍ͯΔʢΫϨΧ৘ใྲྀग़ʁʣ • srv7: όφʔͷը૾͕ϋοΩϯάޙͷը૾ʹͳ͍ͬͯΔ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 27
  28. ো֐ใࠂνϟϯωϧͷ༷ࢠ

  29. Hardening ʹ͓͚Δ ো֐ରԠͷϙΠϯτ

  30. Hardening ʹ͓͚Δো֐ରԠͷϙΠϯτ • ։࢝લͱ։࢝௚ޙͷ४උ͕େ੾ • ಛʹҎԼͷ2఺͕ॏཁ • ໾ׂ෼୲ • ҟৗݕ஌

    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 30
  31. ໾ׂ෼୲ • ΠϯγσϯτίϚϯμʔͷઃஔ • ݕ஌ͱରԠͷ෼཭ • ࣗνʔϜͰ͸ɺγεςϜʹ͸೔ʑ৮Ε͍ͯΔ͚Ͳো֐ରԠͷ ܦݧ͸͋·Γͳ͍ํ͕ͨͪݕ஌໾Ͱͨ͠ 2020/09/04-05 Hardening

    2020 Deep Digital Dependence | @nari_ex 31
  32. ҟৗݕ஌ • ඞͣ؂ࢹγεςϜͱਓͷ໨ͷ྆ํͰߦ͏ • ಥ؏ͰೖΕͨ؂ࢹઃఆ͸࿙Ε͕ଟ͍ͷͰɺਓ͕αʔϏε΍ε ίΞΛఆظతʹ֬ೝ͢Δͷ͸ॏཁ • ҟৗͷ࿈བྷ͸Ͱ͖Δ͚ͩه࿥͕࢒ΔܗࣜͰߦ͏ • ޱ಄ͩͱ৘ใ͕شൃͯ͠๨Εͯ͠·͏ͨΊ

    2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 32
  33. ·ͱΊ • SRE ΍ IMS ΛݩʹτϥϒϧγϡʔςΟϯά΍ΠϯγσϯτϚω δϝϯτͷ஌ݟΛ঺հ͠·ͨ͠ • Hardening Ͱ࣮ࡍʹߦͬͨ͜ͱΛ঺հ͠ɺϙΠϯτΛ·ͱΊ·

    ͨ͠ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 33