Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
5years-history-of-heteml
Shinya Tsunematsu
November 12, 2013
Technology
2
510
5years-history-of-heteml
運用しているヘテムルでのサービス改善の歴史
Shinya Tsunematsu
November 12, 2013
Tweet
Share
More Decks by Shinya Tsunematsu
See All by Shinya Tsunematsu
tnmt
19
7.5k
tnmt
4
1.7k
tnmt
7
1k
tnmt
0
780
tnmt
0
150
tnmt
1
700
tnmt
8
11k
tnmt
0
310
tnmt
9
7.8k
Other Decks in Technology
See All in Technology
ocise
1
1.8k
picardparis
4
2.3k
futo23
1
270
tomihisa
1
1.1k
whitefox_73
0
180
xecus
0
170
eayedi
1
110
ihcomega56
1
440
ryusa
1
190
viva_tweet_x
1
370
manuelmeyer
0
120
miura55
0
240
Featured
See All Featured
kastner
54
1.9k
destraynor
223
47k
michaelherold
224
8.5k
samanthasiow
56
6.3k
hursman
107
9.2k
chriscoyier
780
240k
eileencodes
113
25k
orderedlist
PRO
328
36k
jcasabona
8
550
skipperchong
8
710
62gerente
587
200k
trishagee
23
2.4k
Transcript
2013/11/9 Shinya Tsunematsu @tnmt ͱαʔϏεͷ5ͷาΈ Πϯϑϥվળͷྺ࢙
5ଓ͘αʔϏεͷӡ༻ͬͯ
͘ଓ͘αʔϏεʹ͖ͭ·ͱ͏ ✤ ࢁ͋ΔαʔόؒͰͷඍົͳ༷ࠩ! ✤ ن͕େ͖͍߹ͷมߋͷखؒ! ✤ ΦϯϓϨϛεͷ߹ͷεϖοΫΞοϓʹ͍͍͚ͭͯͳ͍ॊೈੑͷແ ͞! ✤ ϧʔνϯԽ͞Εͨ࡞ۀ͕লྗ͞Ε͍ͯͳ͍͜ͱʹΑΔӡ༻ίετ૿!
✤ Կ͕ى͖͍ͯΔ͔Ѳग़དྷ͍ͯͳ͍αʔϏεϔϧενΣοΫ࣭ͷ ྼԽ
paperboy&co. ϔςϜϧνʔϜ
None
heteml (ϔςϜϧ) ✤ ϩϦϙοϓʂͱฒͿϨϯλϧαʔόͷϥΠϯφοϓ! ✤ 2005ϦϦʔε! ✤ 201310݄ݱࡏαʔό400ऑ! ✤ ৗদ2009͔ΒδϣΠϯ
ٕज़తෛ࠴
ෛ࠴ͱ͍͏ͱωΨςΟϒͳҹʁ ✤ ͔͠͠ඞͣਵ͢ΔͰࣗવͳͷ! ✤ ະવʹ͙ɺൃੜͨ͠߹ʹରॲ͍ͯ͘͠ͷΛ͓Ζ ͔ͦʹ͢ΔͱޙʑࣗͱपΓʹৼΓ͔͔Δ
ΞδΣϯμ ✤ ఏڙ͍ͯ͠ΔαʔϏεͯ͢όʔδϣϯཧԽͰཧ͢Δ! ✤ σϓϩΠαʔόʹର͢ΔมߋπʔϧΛඋɺଐਓԽࡶ͞ Λഉআ! ✤ OSηοτΞοϓΛলྗԽ! ✤ γεςϜߏཧπʔϧΛ༻͍ͯαʔόߏஙͷলྗԽɾ༷Խ
Λߦ͏! ✤ ࢹγεςϜઃఆͷϨϏϡʔɺࢹઃఆͷࣗಈੜ
ͯ͢ΛόʔδϣϯཧԼʹ
ԿΛόʔδϣϯཧ͢Δ͔ ✤ جຊతʹʮͯ͢ʯ
ͱ͍͍͖͑ͳΓશ෦ແཧ ✤ ༏ઌΛ͚ͭΔ! ✤ OSϨΠϠɺηΩϡϦςΟपΓ! ✤ ϛυϧΣΞઃఆ! ✤ cron, ӡ༻εΫϦϓτ!
✤ ࢹιϑτΣΞઃఆ
ΑΓԼͷϨΠϠ͔Βཧ͢Δ ✤ ҙຯͷॏෳΛආ͚ΒΕΔͨΊͦͷ͋ͱͷϝϯςφϯε ੑߴ͘ͳΔ! ✤ ϛυϧΣΞͷηΩϡΞͳઃఆΛܾఆ͢Δ͜ͱ͕༰қ ʹͳΔ! ✤ ҙ֎ͱ͜Ε͕͞Εͯͳ͍έʔε͕ଟ͍
࠷ॳશવཧग़དྷͯͳ͔ͬͨ… ✤ ߏங࣌ظʹΑͬͯઃఆ͕ҧ͏! ✤ ͋Γ·ͤΜ͔ʁ! ✤ iptablesͱtcp_wrapperͰಉ͡ϧʔϧॻ͍ͯΔ! ✤ ϦϓϨΠεΛܦͯɺଘࡏ͍ͯ͠ͳ͍ϗετ͔ ΒͷڐՄઃఆ
όʔδϣϯཧԽʹೖΕ༷Խ ✤ Ұ୴ݱߦಈ͍͍ͯΔαʔόϕʔεͰͱΓ͋͑ͣϦϙ δτϦʹಥͬࠐΉ! ✤ ·͍͠ઃఆͷݟͦ͠ͷޙཧ͢Δ! ✤ มߋτϨʔεՄೳʹͳΔͨΊޙͰݟฦͤΔ
ΦϖϨʔγϣϯπʔϧͷݟ͠
ෳαʔόʹมߋΛՃ͑Δखஈ ✤ όʔδϣϯཧ͕ग़དྷͨΒͦΕΛల։͢Δज़͕ཁΔ! ✤ ͕૿͑Δͱมߋͷख͕ؒେม! ✤ ख͕ؒ૿͑Δͱαʔόؒͷဃ͕૿͑Δѱ॥
sshϧʔϓظ ✤ ຖճଧͭͷ໘ɺઈରϛε͢Δ! ✤ 1ຖ࣮ߦͰγʔέϯγϟϧɺ͕͔͔࣌ؒΔ for host in web{1..100}.heteml.lan;! do!
echo ${host}! ssh ${host} ‘perl -pi -e “s/foo/var/g” /path/to/file’! done
Archerظ ✤ PerlͷσϓϩΠπʔϧ! ✤ YAMLͰఆٛ͞Εͨઃఆʹͱ͖ͮγΣϧίϚϯυ ϑΝΠϧͷσϓϩΠ͕ߦ͑Δ
Archerظ ✤ Pluggable! ✤ Archer::Plugin::RoleLoader! ✤ roles.yaml! ✤ ࢦఆͨ͠ϩʔϧʹ! ✤
ίϚϯυ࣮ߦ! ✤ ϑΝΠϧσϓϩΠ …! ! web:! - web001.heteml.lan! - web002.heteml.lan! mail:! - mail001.heteml.lan! - mail002.heteml.lan! - mail003.heteml.lan! ! …
Archerظ ✤ େྔϗετʹͳΔͱίϚϯυ࣮ߦ݁Ռͷ֬ೝ͕͠ΜͲ ͍! ✤ جຊతʹઃఆϑΝΠϧʹॻ͍ͨ༰Λ࣮ߦ͢ΔͨΊɺ ࡉ͔ͳಈత࡞ۀͷ༻్ʹ͙ͦΘͳ͔ͬͨ! ✤ ಛʹ2ͭΊͷཧ༝͕େมɺಥൃ͢ΔλεΫʹରԠ͖͠ Εͳ͍
ࣗ࡞πʔϧheteloyظ ✤ ༻్ಛԽʢίϚϯυ࣮ߦɺϑΝΠϧσϓϩΠʣ! ✤ ࣗͷPerlπʔϧ! ✤ ʮϔςϜϧʯʹʮσϓϩΠʯ! ✤ σϓϩΠपΓޙड़͢ΔPuppetͱͷੑΛߴΊͨ! ✤
PuppetͷϦϙδτϦͰཧ͞Ε͍ͯΔϑΝΠϧΛ͠ ͘͢ͳ͍ͬͯΔ
ࣗ࡞πʔϧheteloyظ ✤ /var/lib/puppet/data/ ͕ PuppetϦϙδτϦͱͯ͋͠Δ! ✤ /var/lib/puppet/data/modules/web/files/etc/httpd/httpd.conf Λ! ✤ webϩʔϧʢલड़ͷroles.yaml)
ͷϗετʹ͠! ✤ Apache ͷ configtest ͯ͠ OK ͳ߹͚ͩ restart ɺNG ͳ߹ϑΝΠϧϩʔϧόο Ϋ sudo heteloy.pl --role web \! --dist /etc/httpd/httpd.conf \! --mode 644 \! --exec ‘service httpd configtest’ \! --exec ‘service httpd restart’!
ࣗ࡞πʔϧheteloyظ ✤ Α࣮͘ߦ͢ΔλεΫλεΫΛyamlܗࣜʹdump! ✤ Ҏ߱ͦͷyamlΛࢦఆͯ͠ಉ͡ॲཧΛ࣮ߦՄೳ sudo heteloy.pl --role web \!
! --dist /etc/httpd/httpd.conf \! --mode 644 \! --exec ‘service httpd configtest’ \! --exec ‘service httpd restart’ \! --configdump > httpd_task.yaml! sudo heteloy.pl --configloada httpd_task.yaml!
ࣗ࡞πʔϧheteloyظ ✤ ڥґଘΛڐ༰͠൚༻ੑΛࣺͯͨ! ✤ ͷͰެ։ͯ͠ͳ͍Ͱ͢…! ✤ ʑͷӡ༻ͷதͰߦ͏ཻখ͍͕͞εϐʔυΛٻΊ ΒΕΔ࡞ۀʹ͍͍ͯΔ! ✤ PuppetͰͷߏཧ࣠ʹͦΕΛαϙʔτ͢Δҝͷ
πʔϧ͕͋Δͱศར
OSηοτΞοϓࣗಈԽ
৽͍͠ϕϯμͷαʔόΛߪೖ ✤ εϖοΫΞοϓ͕ओ؟! ✤ औΓճ͕͍͢͠͠Α͏IPMIΛࡌ͠ɺԕִʢΦ ϑΟεʣ͔Βͷ࡞ۀ͕ߦ͑ΔΑ͏ʹͨ͠
OSͷΠϯετʔϧDCͤͩͬͨ ✤ IPMI͕͑ΔΑ͏ʹͳΓɺࣗͨͪͰΦϖϨʔγϣϯ Մೳʹ! ✤ ͱ͍͑ɺҰͣͭΠϯετʔϧ͢Δͷେมख͕ؒ ͔͔Δ
Cobbler ✤ http://cobbler.github.com! ✤ ωοτϫʔΫϒʔτͰOSΛΠϯετʔϧ͢Δࡍʹඞཁ ͳίϯϙʔωϯτ͕ͦΖ͍ͬͯΔ! ✤ ݸผͷαʔόͷઃఆใΛதԝͰूதཧՄೳ! ✤ RedHat
Enterprise Linux͓ΑͼͦͷΫϩʔϯOSͰར ༻Մೳ
Cobbler ✤ ૉͷPXEϒʔτͰؤுΕͳ͘ͳ͍͕! ✤ Cobblerdhcpd, pxe, tftp, httpdͳͲΛݸผʹηοτ Ξοϓ͢Δख͕ؒେল͚Δ! ✤
ूதཧ͕༏लɺαʔόઃఆใ (system) ͰIPॏෳ͕ ͋ΔͱΤϥʔʹͳΔͳͲ
CobblerͱPuppetͷॅΈ͚ ✤ CobblerςϯϓϨʔτεχϖοτ͕͑Δ! ✤ ͜ΕΒΛར༻ͯ͠ɺෳϩʔϧ͋ΔαʔόΛΠϯετʔ ϧྃޙଈαʔϏεೖͱ͍ͬͨ͜ͱՄೳͩΖ͏! ✤ kickstartϑΝΠϧϝϯς͠ʹ͍͘! ✤ OSΠϯετʔϧ͑͞ग़དྷΕޙPuppetʹͤΔ
Cobbler ✤ ϗετՃ࣌ɺ͜ͷॲཧޙαʔόΛىಈ͢ΔͱOS ͕Πϯετʔϧ͞ΕΔ ʢωοτϫʔΫ෦ׂѪʣ cobbler system add \! --name
web001.heteml.lan \! --hostname web001.heteml.lan \! --profile CentOS6.4-x86_64 \! --kopts “ksdevide=XX:XX:XX:XX:XX syslog=192.168.X.X” \! --kickstart /var/lib/cobbler/kickstarts/CentOS6-base.ks! ! cobbler sync
γεςϜߏཧπʔϧͷಋೖ
ಋೖͷඞཁੑ ✤ αʔόಋೖϖʔε͕Ճ! ✤ ߏஙͷख͕ؒϘτϧωοΫʹ
طଘαʔϏεͷPuppetಋೖ ✤ खॱॻΛಡΈղ͘! ✤ ඞཁͳ߹खॱॻͷཧ! ✤ ͖ͪΜͱඥղ͖ɺߏཧπʔϧͷઃఆϑΝΠϧʹམͱ ͨ࣌͠ʹͦΕ͕αʔόͷ”༷”ͱͳΔ! ✤ Ұ࣌ظΛࢧ͑ͨൿͷλϨܾͯ͠ѱͷΈͰͳ͍!
✤ ݱ࣮ੈքͷͦΕͱҟͳΔͷ֧፩ʢ͔͘Μʣ͠ͳ͍ҝ
ݕূ͢Δڥ ✤ ࢼͯ͠ഁغͯ͠Γ͕͠ग़དྷΔVMϕʔεͰߦ͏ͷ ͕ΓΓ͍͢! ✤ ϔςϜϧͰϖύϘࣾڥͷKVMཧπʔϧ MaglicaΛͬͯखܰʹVM࡞ɺഁغΛߦͳͬͨ! ✤ ݱࡏͳΒVagrantͳͲ͕͋ΔͨΊͦΕΛར༻͢Δͱ༗ ༻ͦ͏
·ͣ৽نՃϗετ͔Βಋೖ ✤ ৽نϗετطଘαʔόͱͷ͕͠ΒΈ͕ͳ͍! ✤ Ѳ͍ͯ͠ͳ͍༷࿙ΕͳͲ͕͋ΔՄೳੑ͋Γطଘ αʔό৻ॏʹߦ͏ඞཁ͕͋Δ! ✤ طଘϗετͷద༻noop(dry-run)࣮ߦ͠ɺࠩ֬ ೝɺඞཁ͕͋Εमਖ਼ͳͲߦ͍ͭͭॱ࣍ಋೖΛਐΊ ͍ͯΔ
ಋೖ͕ྃͨ͠ޙଓ͘ ✤ ߏཧπʔϧͷهड़ํࣜɺϕετϓϥΫςΟε͕ม ΘΔ͜ͱɺӡ༻͍ͯ͘͠தͰෆཁͳهࡌͳͲग़ͯ ͘Δ! ✤ ࣺͯΔ༐ؾ! ✤ ఆظతʹݟ͠Λ͢Δඞཁ͕͋Δ
ಋೖ͕ྃͨ͠ޙଓ͘ ✤ ඞཁͳઃఆόʔδϣϯཧ͞Ε͍ͯΔ! ✤ ༷ͷมߋݟ͠ઃఆϑΝΠϧϕʔεͰٞ͢Δ! ✤ ϖύϘͰGH:EΛར༻͍ͯ͠ΔͨΊɺPull Requestϕʔ εͰ্هΛߦͳ͍ͬͯΔ! ✤
ϨϏϡʔͯ͠Β͑Δͱਫ਼͕͕͋Δ! ✤ ͋ͱ҆৺͢Δ
ࢹγεςϜͷӡ༻վળ
ࢹ͕ἧͬͯͳ͍ ✤ ࢹ߲͕ࡏΔͷͱແ͍ͷ͕͋Δ! ✤ ϗετΛՃ͢Δ͝ͱʹखՃ! ✤ ͦͦࢹ͞Ε͍ͯͳ͍ϗετ…ʁ
αʔόͷҰཡ࣋ͬͯΔ ✤ લड़ͷroles.yaml! ✤ ͜͜ʹཧ͍ͯ͠Δαʔόཏ͞Ε͍ͯΔ
ࢹઃఆͷࣗಈੜ ✤ Nagios! ✤ munin! ✤ roles.yaml Λͱʹ conf Λࣗಈੜ͢Δ!
✤ શαʔόڞ௨ͷࢹ (σΟεΫI/O, L/A)! ✤ ϩʔϧݻ༗ͷࢹ (webͩͬͨΒHTTPૄ௨)! ✤ ϗετݻ༗ͷઃఆ (ಛఆαʔόαʔό্Ͱύϥϝʔλ্ॻ͖ʣ
มߋ࣌ ✤ طଘͷࢹγεςϜʢखಈՃʣͱ৽ࢹγεςϜʢઃ ఆࣗಈੜʣΛฒߦͯ͠Քಇͤ͞Δ! ✤ Nagios৽ࢹγεςϜΞϥʔτඈ͞ͳ͍! ✤ ेʹ֬ೝ͔ͯ͠Β͍͑ͰΓସ͑
ৼΓฦͬͯ
ྺ࢙͕͍ͷͰτϨϯυมΘΔ ✤ Puppetmaster agentܕͰͬͯΔ͚Ͳɺ࠷ۙελ ϯυΞϩϯܕ͕ྲྀߦͬͯΔʢςετۦಈΠϯϑϥߏஙʣ ✤ SerfͳͲʹΑΔࣗతʢಈతʣͳϊʔυՃɾઃఆ ՃͳͲ! ✤ ͳͲͳͲ
Δ͔͠ͳ͍ ✤ WebͷαʔϏεͳݶΓ͕ͳ͍ɺӡ༻ʹऴΘΓ͕ ͳ͍! ✤ ͚Ͳޮతʹɺָ͘͢͠Δ͜ͱग़དྷΔ! ✤ Ұ൪ϘτϧωοΫʢࡶɺ͕͔͔࣌ؒΔʣ࡞ۀ͔Βૣ ΊʹऔΓΉ! ✤
ಓʹΔ
ྫ ✤ ࠷ۙϝϯς͕ଟ͍! ✤ ରϗετ͕ҧ͏͚ͩͰखॱ͍ಓ͕ฉ͘! ✤ खॱॻੜʢςϯϓϨʔτԽʣ͠Α͏ʂ
खॱΛςϯϓϨʔτԽ export OLDHOST=<%= @config['src1_hostname'] %>! export NEWHOST=<%= @config['dst_lan_ip'] %>! rsync
-rlptDv /data/backup/heteml/$OLDHOST/path/from/ $NEWHOST::data_to/ src1_hostname : web001! src1_lan_ip : 192.168.1.10! ! dst_hostname : web002! dst_lan_ip : 192.168.1.11!
खॱΛςϯϓϨʔτԽ genecon -t README.md -c config.yaml > web001maintenance.md export OLDHOST=web001!
export NEWHOST=192.168.1.11! rsync -rlptDv /data/backup/heteml/$OLDHOST/path/from/ $NEWHOST::data_to/ ✤ genecon https://github.com/glidenote/genecon
͋Δఔͷڳඞཁ ✤ ৽͍͠ͷͷಋೖԿ͔ͷ৽༐ؾ͕ཁΔ! ✤ ͰΔ࣌ʹΒͳ͍ͱ͍͚ͳ͍! ✤ ཪ͚Ͱ͋Δ֬ೝςετͳͲͰཪ͚Λͯ҆͠৺ ͢Δ
ࠓճͷ༰ɺ͞Βʹৄ͍͠༰ ✤ WEB+DB PRESS Vol.75! ✤ ʙෳࡶੑͷ૿େͱڥͷมԽʹରԠ͢Δʙ ܧଓతWebαʔϏεվ ળΨΠυ! ✤
ΞϓϦɺνʔϜϏϧσΟϯάͷվળͳͲଞͷτϐοΫ߹Θͤͯࡌͬ ͯ·͢
Ұॹʹಇ͘ϝϯόʔΛืू͍ͯ͠· ͢ ✤ http://www.paperboy.co.jp/recruit/engineer_career/! ✤ ϔςϜϧνʔϜͷΠϯϑϥΤϯδχΞืूதͰ͢
͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠