Hardening におけるトラブルシューティング / Troubleshooting in Hardening

Hardening 2020 Deep Digital Dependence の登壇資料です。
https://wasforum.jp/2020/08/hardening-2020-deep-digital-dependence/

1. Hardening ʹ͓͚Δ τϥϒϧγϡʔςΟϯά גࣜձࣾϋʔτϏʔπ औక໾ ߴଜ ੒ಓ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex
2. about me • ॴଐ • גࣜձࣾϋʔτϏʔπ औక໾ • ޷͖ͳήʔϜ • ετϦʔτϑΝΠλʔV • Hardening • 2020 BO ͰάϥϯϓϦड৆ • ΑΓৄࡉͳ৘ใ͸ͪ͜Β΁ • h1ps:/ /www.nari-ex.com/about/ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 2
3. גࣜձࣾϋʔτϏʔπ • 2005೥૑ۀ • ࣄۀ಺༰ • MSP ࣄۀ • Πϯϑϥӡ༻ͷΞ΢τιʔαʔ • ؂ࢹɺઃܭɺߏஙɺΫϥ΢υಋೖࢧ ԉɺίϯαϧςΟϯά • ։ൃࣄۀ • ࣾһ਺: 71໊ʢ໿8ׂҎ্͕ΤϯδχΞʣ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 3
4. ͜ͷηογϣϯʹ͍ͭͯ • Hardening ࣗମͷઆ໌͸ׂѪ͠·͢ • Hardening ͷো֐ରԠΛߦ͏্ͰॏཁͳϙΠϯτΛ͓࿩͠·͢ • τϥϒϧγϡʔςΟϯάະܦݧͷํ޲͚ͷࢿྉͰ͢ • ۩ମతͳରԠৄࡉʢઃఆ΍ίϚϯυͳͲʣͷ࿩͸͠·ͤΜ • SRE ΍ IMS ͷ஌ݟ΋౿·͑ͯൃද͠·͢ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 4
5. Agenda • SRE ͱ IMS ͔Βͷֶͼ • τϥϒϧγϡʔςΟϯά • ΠϯγσϯτϚωδϝϯτ • Hardening BO 2020 Ͱ࣮ࡍʹ΍ͬͨ͜ͱ • Hardening ʹ͓͚Δো֐ରԠͷϙΠϯτ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 5
6. τϥϒϧγϡʔςΟϯά
7. Site Reliability Engineering 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 7
8. SREຊʹ͓͚Δো֐ରԠʹؔ࿈͢Δষ • 12ষ: ޮ཰తͳτϥϒϧγϡʔςΟϯά • 13ষ: ۓٸରԠ • 14ষ: Πϯγσϯτ؅ཧ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 8
9. ෺ࣄ͕͏·͍͘͘৔߹ͱ͍͏΋ͷ ͸ɺ෺ࣄ͕͓͔͘͠ͳΔ৔߹ͷதͷ ಛघͳྫʹա͗ͳ͍ɻ — John Allspaw 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 9
10. τϥϒϧγϡʔςΟϯάͷϓ ϩηε • ໰୊ͷϨϙʔτ • ظ଴͞ΕΔಈ࡞ͱ࣮ࡍͷಈ࡞Λࣔ͢ • ex. ࣗಈԽ͞Εͨ؂ࢹγεςϜʹΑΔΞϥʔτ௨஌ • ex. ಉ྅͔ΒʮγεςϜ͕஗͘ͳ͍ͬͯΔʯͱฉ͍ͨ • τϦΞʔδ • ໰୊ͷॏཁ౓Λ൑அ্ͨ͠ͰͳʹΛ͢΂͖͔൑அ͢Δ • ؍࡯ɺ਍அ • ର৅ͷγεςϜͷ৘ใΛݩʹݪҼΛಛఆ͢Δ • ςετ/ରॲɺ෮چ • ಛఆͨ͠ݪҼ΁ͷରࡦΛߦ͍ɺ෮چͤ͞Δ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 10
11. τϥϒϧγϡʔςΟϯάͷϓϩηεΛཧղ͢Δ͚ͩͰ͸ෆे෼ Hardening BO 2020 Ͱ͸͜Μͳ؀ڥͰͨ͠ • ॳݟ ͷγεςϜ܈ • 6ʙ8໊ఔ౓ͷνʔϜ Ͱ࡞ۀΛߦ͏ • ސ٬ͱͷίϛϡχέʔγϣϯ ͕ੜ͡Δ • ෺ཧతʹ෼཭ ͨ͠ΦϖϨʔγϣϯࣨ • ۮવى͖Δো֐Ͱ͸ͳ͘ɺഁյΛ໨తͱͨ͠߈ܸ΁ͷରԠ • ...etc 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 11
12. Hardening ͷτϥϒϧγϡʔςΟϯά • ຖ೔৮ͬͯΔγεςϜͷখ͞ͳো֐Λࣗ෼ҰਓͰαΫοͱରԠ ͢ΔΑ͏ͳ΋ͷͰ͸ͳ͍ • → ໨ͷલͷγεςϜ͚ͩͰͳ͘ɺো֐ରԠʹཱͪ޲͔͏ਓͷ໾ ׂ΍ମ੍΋౿·͑ͯߟ͑Δ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 12
13. ΠϯγσϯτϚωδϝϯτ
14. “Fire is not an emergency to the fire department. It’s what we do.” Րࡂ͸ফ๷ॺʹͱͬͯۓٸࣄଶͰ͸ ͳ͍ɻͦΕ͸զʑͷ΍Δ͜ͱͩɻ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 14
15. Incident Management for Opera2ons • ফ๷΍ॏཁΠϯϑϥʹؔΘ͖ͬͯͨਓ͕ͨͪ IMS Λ IT ӡ༻ʹ ద༻ͤͨ͞΋ͷΛ঺հ͍ͯ͠Δ • IMS ͱ͸ɺ΋ͱ΋ͱ40೥Ҏ্ʹ౉Γɺશถͷফ๷ॺ͕͋ΒΏΔ छྨͷۓٸࣄଶʹରԠ͢ΔͨΊʹར༻͖ͯͨ͠ϑϨʔϜϫʔΫ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 15
16. ࢀߟ: ΠϯγσϯτϥΠϑαΠΫϧ ΠϯγσϯτରԠͷମ੍΋౿·͑ͯରԠΛߦ͏ 1. ໰୊Λݕ஌͢Δ 2. ໰୊͕Πϯγσϯτͳͷ͔Πϕϯτͳͷ͔൑அ͢Δ 3. ద੾ͳΠϯγσϯτରԠνʔϜΛฤ੒͢Δ 4. ରԠνʔϜͷฏۉ૊Έཱͯ࣌ؒʢMTTAʣΛ૊ΈཱͯΔ 5. ΠϯγσϯτίϚϯμʔΛཱ֬͠ɺϦιʔεΛ੔ཧ͠ɺΠϯγσϯτͷ໨తΛઃఆ͢Δ 6. IMSΛར༻ͯ͠ɺରԠΛߦ͏ 7. ద੾ͳར֐ؔ܎ऀʹ௨஌͢Δ 8. ΠϯγσϯτΛղܾͯ͠ɺϦιʔεΛղ์͢Δ 9. AARʢA1er Ac6on ReviewʣΛ࣮ࢪ͢Δ 10. ඼࣭վળʢQIʣͱ඼࣭อূʢQAʣΛ࣮ࢪ͢Δ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 16
17. ΠϯγσϯτରԠνʔϜ • Incident Commander (IC) • ΠϯγσϯτରԠͷϦʔμʔ • ҙࢥܾఆɺϑΝγϦςʔλʔ • Communica2on Officer (Comms) • ICΛิࠤ͠ɺؔ܎ऀʹ࿈བྷΛߦ͏ਓ • Situa2onal Status (Scribe) • Πϯγσϯτʹؔ࿈ͨ͠৘ใΛه࿥͢Δਓʢॻهʣ • Group Leader • ઐ໳ՈάϧʔϓͷϦʔμʔɻ࣮ࡍʹରԠΛऔΓ࢓੾Δਓ • Subject Ma>er Expert (SME) • ઐ໳Ո 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 17
18. ͜͜·Ͱͷ·ͱΊ • τϥϒϧγϡʔςΟϯά • ໰୊ͷϨϙʔτ͕ى఺ • ԾઆɾݕূΛ܁Γฦ͢͜ͱͰγεςϜΛ෮چʹಋ͘ • ΠϯγσϯτϚωδϝϯτ • ΠϯγσϯτίϚϯμʔ͕ରԠͷத৺ • ໾ׂ͸ணखલʹܾΊΔ • ࣮ରԠ͢Δ໾ׂʢSMEʣҎ֎΋ׂͱ͋Δ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 18
19. Hardening 2020 BO Ͱ ࣮ࡍʹ΍ͬͨ͜ͱ
20. ͜Ε·Ͱͷ஌ݟΛࣄલ४උʹద༻ • γεςϜҎ֎ͷ୲౰΋ؚΊͯ໾ׂ෼୲Λߦ͏ • ΠϯγσϯτίϚϯμʔΛ໌֬ʹ͢Δ • ҟৗݕ஌ͷ࢓૊ΈΛߟ͑Δ • ؂ࢹγεςϜʹΑΔݕ஌ • ਓʹΑΔݕ஌ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 20
21. લ೔ ·Ͱ ʹ΍ͬͨ͜ͱ • ໾ׂͷΞαΠϯ • શһͰإ߹ΘͤʢҿΈձʣ • τϥϒϧγϡʔςΟϯά୲౰಺Ͱͷଧͪ߹Θͤ • γεςϜ୲౰͸ߏ੒೺Ѳ • ҰਓͻͱΓ͕ॳಈ࣌ʹ΍Δ͜ͱΛܾఆ • αʔϏεURLҰཡͷ४උ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 21
22. ໾ׂ෼୲ͷ༷ࢠ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 22
23. ໾ׂ෼୲ͷ಺༰ • ΠϯγσϯτίϚϯμʔΛࣄલʹܾఆ && ݕ஌ͱରԠͷ໾ׂΛ۠ผ • ۩ମతʹ͸ҎԼͷ5ͭͷ໾ׂΛׂΓ౰ͯ • ΠϯγσϯτίϚϯμʔʢλεΫ༏ઌ౓ɺܦࡁ৚݅ͷҙࢥܾఆͳͲʣ • τϥϒϧγϡʔςΟϯά1ʢݕ஌ɺ੾Γ෼͚ʣ • τϥϒϧγϡʔςΟϯά2ʢରࡦ࣮ࢪʣ • ಺෦΁ͷϨϙʔλʔ݉ব֎ • ൢച؅ཧʢαʔϏεʹΑΔച্ͷ؅ཧͳͲʣ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 23
24. ։࢝௚ޙʹ΍ͬͨ͜ͱ • ؂ࢹπʔϧʢZabbixʣ ʹΑΔ؂ࢹઃఆ • ར༻͍ͯ͠ͳ͍ϛυϧ΢ΣΞͷఀࢭɾ࡟আ • Rainloop, phpmyadmin, phppgmyadmin ͳͲ • ֤ॴͷύεϫʔυมߋ • ֤σʔλͷόοΫΞοϓ • ෆཁͳωοτϫʔΫΞΫηεͷःஅ • ex. WAN ͔Β DMZ ΁ͷ DB ϙʔτ΁ͷΞΫηεःஅ • IPS ͷಋೖʢCisco ༷ఏڙʣ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 24
25. ౰೔ͷτϥϒϧγϡʔςΟϯά • Өڹൣғͷେ͖͍ো֐Λ༏ઌతʹରॲ • ඞཁʹԠͯ͡ো֐ରԠؒͰϖΞΦϖ • ணखͱ෮چͷ࿈བྷ͸͜·Ίʹ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 25
26. ࢀߟ: ։࢝࣌ͷෆඋɾෆ۩߹ʹΑΔো֐ • ࠾༻αΠτ: DB ઀ଓઌͷ࠶ઃఆ • νϟοτ: Docker ίϯςφΛਖ਼͘͠ىಈ • Shop3: ࣄલ഑෍ͷυΩϡϝϯτͷURL͕͍ؒͬͯΔ • Shop3: POST ϦΫΤετ࣌ͷϦΫΤεταζ্ݶ͕10KͰ͋ͬͨͨΊ͕Ҿ্͖͛ • m-pay: υϝΠϯ໊Λਖ਼͍͠΋ͷʹSQLʹ͖ͯ׵͑ • m-pay: ϓϥάΠϯϑΝΠϧͷΞΫηεݖݶमਖ਼ • Nginx: Nginx + PHP-FPM +fastcgi_split_path ͷ૊Έ߹ΘͤʹΑΔऑੑͷमਖ਼ • EC-CUBE: ΠϯετʔϧεΫϦϓτͷୀආ • EC-CUBE: େྲྀߦͨ͠ط஌ͷ੬ऑੑ΁ͷରࡦ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 26
27. ࢀߟ: ։࢝ޙͷ߈ܸʹΑΔো֐ • Shop: iptables ͷ INPUT νΣΠϯʹ443, 80 ϙʔτ΁ͷ઀ଓΛ DROP ͢Δϧʔϧ௥Ճ͞Ε͍ͯ Δ • શମ: αʔό಺Ͱ໊લղܾ͕Ͱ͖ͳ͍ • srv4,6: ΫϨδοτΧʔυ৘ใΛҾͬ͜ൈ͘JS ϑΝΠϧ͕ஔ͔Ε͍ͯΔ • શମ: ϛυϧ΢ΣΞ͕஌Β͵ؒʹམ͍ͪͯΔ • Shop: PHP-FPM ͕ਖ਼ৗʹಈ࡞͠ͳ͍ • srv4,6: 25൪ϙʔτΛར༻ͯ͠֎෦΁େྔσʔλ͕ૹ৴͞Ε͍ͯΔʢΫϨΧ৘ใྲྀग़ʁʣ • srv7: όφʔͷը૾͕ϋοΩϯάޙͷը૾ʹͳ͍ͬͯΔ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 27
28. ো֐ใࠂνϟϯωϧͷ༷ࢠ
29. Hardening ʹ͓͚Δ ো֐ରԠͷϙΠϯτ
30. Hardening ʹ͓͚Δো֐ରԠͷϙΠϯτ • ։࢝લͱ։࢝௚ޙͷ४උ͕େ੾ • ಛʹҎԼͷ2఺͕ॏཁ • ໾ׂ෼୲ • ҟৗݕ஌ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 30
31. ໾ׂ෼୲ • ΠϯγσϯτίϚϯμʔͷઃஔ • ݕ஌ͱରԠͷ෼཭ • ࣗνʔϜͰ͸ɺγεςϜʹ͸೔ʑ৮Ε͍ͯΔ͚Ͳো֐ରԠͷ ܦݧ͸͋·Γͳ͍ํ͕ͨͪݕ஌໾Ͱͨ͠ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 31
32. ҟৗݕ஌ • ඞͣ؂ࢹγεςϜͱਓͷ໨ͷ྆ํͰߦ͏ • ಥ؏ͰೖΕͨ؂ࢹઃఆ͸࿙Ε͕ଟ͍ͷͰɺਓ͕αʔϏε΍ε ίΞΛఆظతʹ֬ೝ͢Δͷ͸ॏཁ • ҟৗͷ࿈བྷ͸Ͱ͖Δ͚ͩه࿥͕࢒ΔܗࣜͰߦ͏ • ޱ಄ͩͱ৘ใ͕شൃͯ͠๨Εͯ͠·͏ͨΊ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 32
33. ·ͱΊ • SRE ΍ IMS ΛݩʹτϥϒϧγϡʔςΟϯά΍ΠϯγσϯτϚω δϝϯτͷ஌ݟΛ঺հ͠·ͨ͠ • Hardening Ͱ࣮ࡍʹߦͬͨ͜ͱΛ঺հ͠ɺϙΠϯτΛ·ͱΊ· ͨ͠ 2020/09/04-05 Hardening 2020 Deep Digital Dependence | @nari_ex 33