Goでサーバの健全性を確保する / Keeping servers healthy with Go

FUKUOKA Engineers Day 2021 Online
Tomohisa Oda / Apr 24, 2021
Keeping Servers Healthywith Go
GoͰαʔόͷ݈શੑΛ֬อ͢Δ

View Slide

@linyows
Blog: tomohisaoda.com

Affiliation: GMO pepabo, inc.

Fukuoka.go, Fukuoka.ts Organizer
࠷ۙɺNotionʹࣄྫ঺հهࣄͰࣅإֆΛॻ͍ͯ΋Β͍·ͨ͠ɻ

ࠓ೔͸ͳΜͱɺGo Conference Tokyo 2021͕ࠓ͍͋ͬͯ·͢ɻ#gocon

View Slide

๏ ϗεςΟϯάࣄۀ෦ Senior Engineering Lead

ࣄۀ෦ͷCTO΍VPoEͷΑ͏ͳ΋ͷ

๏ ϖύϘݚڀॴ ۝भେֶڞಉݚڀ

2020 Պݚඅ࠾୒ https://kaken.nii.ac.jp/grant/KAKENHI-PROJECT-20K11791/

ܰྔίϯςφʹΑΔେن໛ߴूੵϝʔϧϗεςΟϯάج൫ʹ͓͚Δૹ৴ػೳͷߴػೳԽ
In Pepabo…

View Slide

Private Projects
ΞϓϦέʔγϣϯࣗ਎͕ϦϙδτϦ͔
Β࠷৽ͷίʔυΛݕ஌ͯࣗ͠ಈͰσϓϩ
ΠͱσϦόϦʔΛߦ͏πʔϧ
linuxͷuser໊લղܾΛGitHubͷ
Teamϝϯόʔ΍ϦϙδτϦͷΞΫηε
ݖݶऀ͔Βߦ͏πʔϧ
ϝʔϧૹ৴༻ͷಁաܕSMTPϓϩΩγͰ
ϦΫΤετͱϨεϙϯεΛϩά΍σʔλ
ϕʔεʹग़ྗ͢Δ͜ͱ͕Ͱ͖Δ
New!!

View Slide

GoͰαʔόͷ݈શੑΛ֬อ͢Δ 👨⚕

View Slide

๏ ϜʔϜʔυϝΠϯ — https://muumuu-domain.com

๏ ΁ςϜϧ — https://heteml.jp

๏ ϩϦϙοϓ — https://lolipop.jp

๏ ϚωʔδυΫϥ΢υ — https://mc.lolipop.jp
ϗεςΟϯάࣄۀ෦

View Slide

๏ Domain — DNS

๏ Compute — HTTP, SSH, FTP and WebDav…

๏ Mail — SMTP, POP, IMAP…

๏ Database — MySQL, SQLite…
֤αʔόϦιʔεͷ҆ఆతఏڙͱӡ༻

View Slide

αʔόΛ؅ཧ͢Δ͜ͱ͕ͱͯ΋ॏཁ ⚠

View Slide

෺ཧαʔόͷ؅ཧ
๏ ϗεςΟϯάࣄۀ෦͚ͩͰ෺ཧαʔό͕େମ1000୆Ҏ্͋Δ

๏ αʔό͸ނো͢Δ΋ͷ

๏ อकظؒ͸௕ͯ͘5೥ؒɺյΕͨΒഇغ͢Δ͔ɺߴֹͳඅ༻Λ෷ͬͯमཧ͢Δ͔

๏ αʔόͷݮՁঈ٫͸5೥ɺͭ·Γαʔόͷ଱༻೥਺͕লྩͰ5೥ͱͳ͍ͬͯΔ

๏ ఆظతʹαʔόΛϦϓϨʔε͢Δඞཁ͕͋Δ

๏ ͪͳΈʹϕϯμʔʹΑͬͯn೥΋ͭɺ΋ͨͳ͍Έ͍ͨͳͷ͸ͳΜͱͳ͋͘ΔͬΆ͍

View Slide

Ծ૝αʔόͷ؅ཧ
๏ ෺ཧαʔό্ʹ૥ͷKVM΍OpenStackΛ࢖ͬͯԾ૝؀ڥΛ࡞͍ͬͯΔ

๏ ࠷ۙ͸OpenStack্ʹKubernetes؀ڥΛ࡞ͬͯਫ਼ྗతʹίϯςφԽΛਐΊ͍ͯΔ

๏ VM΋ͬ͟ͱ600Πϯελϯε΄Ͳ͋Δ

๏ ౰વϋʔυ΢ΣΞҎ֎ʹιϑτ΢ΣΞ΋ݹ͘ͳΔͷͰɺ੬ऑੑରԠͳͲɺOS΍ϛυϧ
΢ΣΞͷΞοϓσʔτ΋ܧଓతʹඞཁ

๏ ϑΝΠϧͷ੔߹ੑ؂ࢹ΍ڴҖݕग़΍੬ऑੑݕग़ʹWazuhΛར༻͍ͯ͠Δ͕໢ཏతͰ͸ͳ
͍

View Slide

ηΩϡϦςΟͷ؅ཧ
๏ IDSʢ৵ೖݕ஌γεςϜʣ
ʹWazuhΛར༻͍ͯ͠Δ

๏ Wazuh͸ڴҖݕ஌Ҏ֎ʹ
ϑΝΠϧͷ੔߹ੑ؂ࢹ΍
੬ऑੑݕग़ʢCVE
databaseʣ΋Ͱ͖Δ

View Slide

ࢿ࢈ͱͯ͠ͷαʔό؅ཧ
๏ ߴֹͳαʔό͸ձࣾࢿ࢈ͷ̍ͭ

๏ ͍ͭɺͩΕ͕ɺͲ͔͜ΒɺͳʹΛɺͳΜͷͨΊʹɺ͕෼͔ΔΑ͏ʹ؅ཧ͍ͯ͠Δ

๏ ࣗલͷΞϓϦέʔγϣϯͱDBʹΑ࣮ͬͯݱ͍ͯ͠Δ

๏ Ұൠతʹ͸ɺDCIMʢData Center Infrastructure Managementʣ΍IPAMʢIP
Adress ManagerʣͷൣᙝͰ͋Δαʔό͕ऩ༰͞ΕΔϥοΫͷ৔ॴ΍αʔόͷεϖοΫɺ
αʔό͕࢖͍ͬͯΔIPΞυϨεͷ؅ཧ΋ͦͷΞϓϦέʔγϣϯʹೖ͍ͬͯΔ

๏ ࢿ࢈؅ཧ͸ͪΌΜͱͰ͖͍ͯΔ

View Slide

😭 ؅ཧ΍ӡ༻͢Δ͜ͱ͕ΊͬͪΌ͋Δ 😭

View Slide

՝୊
๏ ෳ਺ͷαʔϏεɺෳ਺ͷνʔϜɺෳ਺ͷαʔό؀ڥʹΑΓશମ૾ͷ೺Ѳ͕Ͱ
͖͍ͯͳ͍

👉 ໢ཏతʹશମ೺Ѳ͕Ͱ͖Δঢ়ଶʹ͍ͨ͠ʂ

๏ αʔό؅ཧΛϝΠϯʹ΍͍ͬͯΔνʔϜ͸ͳ͘ӡ༻ͷยखؒͳͨΊɺରԠ͕
ޙखʹͳΓ͕ͪ

👉 ੵۃతʹαʔόͷ݈શੑΛҡ࣋Ͱ͖Δঢ়گΛͭ͘Γ͍ͨʂ

View Slide

GoͰ͋ΒΏΔαʔόͷInventoryΛͭ͘Δ

View Slide

Architecture
- ෺ཧαʔόೲ඼࣌ʹਓ͕ࢿ࢈DBʹొ࿥͢Δ

- ࢿ࢈DB͔Β෺ཧαʔόͷ৘ใऔಘ͢Δ

- KVMϗετʹSSHͯ͠ήετ৘ใΛऔಘ͢Δ

- OpenStack͸API͔Βςφϯτ͝ͱʹ
Instanceͷ৘ใΛऔಘ͢Δ

- શͯΛϦετԽͨ͠ϑΝΠϧΛgit؅ཧͯ͠ຖ
िࠩ෼Λࢦࠩ֬͠ೝ͢Δ

- Wazuhαʔόͱagentͷ໊دͤΛͯ͠શ୆ಋ
ೖΛ࿙Εͳ͘΍Δʢಋೖ཰ͷՄࢹԽʣ

View Slide

Architecture
- ෺ཧαʔόೲ඼࣌ʹਓ͕ࢿ࢈DBʹొ࿥͢Δ

- ࢿ࢈DB͔Β෺ཧαʔόͷ৘ใऔಘ͢Δ

- KVMϗετʹSSHͯ͠ήετ৘ใΛऔಘ͢Δ

- OpenStack͸API͔Βςφϯτ͝ͱʹ
Instanceͷ৘ใΛऔಘ͢Δ

- શͯΛϦετԽͨ͠ϑΝΠϧΛgit؅ཧͯ͠ຖ
िࠩ෼Λࢦࠩ֬͠ೝ͢Δ

- Wazuhαʔόͱagentͷ໊دͤΛͯ͠શ୆ಋ
ೖΛ࿙Εͳ͘΍Δʢಋೖ཰ͷՄࢹԽʣ
0
150
300
450
600
A B C D

View Slide

ग़དྷͨ΋ͷ

View Slide


View Slide

ಈ͘΋ͷ͕Ͱ͖Δ·Ͱ̍िؒ
- ։ൃͨ͠΋ͷΛҰൠԽ͠ʹ͍ͨ͘Ίɺιʔε
ίʔυ͸ඇެ։

- Architecture͔Β෼͔Δ௨Γฐࣾͷӡ༻ମ
੍ґଘ͕େ͖͍

- Go͸ඪ४Ϟδϡʔϧ͕ॆ࣮͍ͯ͠ΔͷͰ֎
෦ϥΠϒϥϦ͸ൺֱతʹগͳ͍

View Slide

Test
- Integrationςετ͸ྗٕͬΆ͍͕ɺSSH
αʔόΛGoͰىಈͤͯ͞Stub͢ΔΑ͏ʹ͠
͍ͯΔ

- ͨͩɺSSHαʔόॻ͘ͷ΋݁ߏखܰͰ170ߦ
͙Β͍Ͱॻ͚Δ

$ wc -l ssh_test.go

170

View Slide

SSHӽ͠ʹTest
- SSHαʔό͕ड͚औͬͨೖྗ͔Βग़ྗ͢Δ
಺༰ΛఆٛΛ͢Δ

- goroutineͰSSHαʔόΛىಈ͢Δ

- ର৅ͷϝιουΛݺͼग़͢

- ظ଴͢ΔίϚϯυΛSSHαʔό͕ड͚औͬ
͍ͯΔ͔Λݕূ͢Δ

View Slide

Metrics
- ύϑΥʔϚϯεϞχλϦϯά͸PrometheusΛ࢖͍ͬͯΔ

- PrometheusͷΧελϜίϨΫλͱͯ͠ϑΝΠϧΛग़ྗ

- ෺ཧαʔόͷՔಇ೥਺Λyoung, middle,oldͱͯ͠෼ྨ͠ɺ
αʔϏε͝ͱͷαʔόͷ݈શੑΛՄࢹԽ͍ͯ͠Δ

- ՄࢹԽ͢Δ͜ͱͰੵۃతͳαʔόϦϓϨʔε͕ՄೳʹͳΔͱߟ
͍͑ͯΔ

View Slide

๏ Ϗδωε্͋ΒΏΔαʔόΛ؅ཧ͍ͯ͘͜͠ͱ͕ॏཁͰ͋Δ

๏ ෺ཧαʔό΍ෳ਺؀ڥͷԾ૝αʔόʹରͯ͠໢ཏతͳ୆ாΛGoΛ࢖ͬͯ࡞੒ͨ͠

๏ ୆ாͷ৘ใ͔Βશαʔό࿙Εͳ͘IDSΛಋೖ͢Δ͜ͱ΍ɺੵۃతͳαʔόϦϓϨʔεΛߦ
͏؀ڥΛ੔උͰ͖ͨ

๏ ΍΍ෳࡶͳߏ੒ͷεΫϦϓτͰ΋ൺֱత؆୯ʹIntegration TestͰ͖Δ
Conclusion

View Slide

Fukuoka.go͸ɺGoݴޠΛத৺ʹιϑτ΢ΣΞΤϯδχΞϦϯάͷڞ༗Λߦ͏ίϛϡ
χςΟͰ͢ɻ2020೥ͷΤϯδχΞϑϨϯυϦʔγςΟ෱ԬΞϫʔυίϛϡχςΟ෦໳
Ͱड৆͠·ͨ͠ɻ

ڵຯ͕͋Γ·ͨ͠ΒɺConnpassͳͲ͔Β͝ࢀՃ͍ͩ͘͞ʂ

https://fukuokago.dev/
Fukuoka.go

View Slide

Thank you \ʕ⊙౪⚆ʔ/

View Slide

Goでサーバの健全性を確保する / Keeping servers healthy with Go

Goでサーバの健全性を確保する / Keeping servers healthy with Go

linyows

More Decks by linyows

Other Decks in Programming

Featured

Transcript