Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
5years-history-of-heteml
Search
Shinya Tsunematsu
November 12, 2013
Technology
2
670
5years-history-of-heteml
運用しているヘテムルでのサービス改善の歴史
Shinya Tsunematsu
November 12, 2013
Tweet
Share
More Decks by Shinya Tsunematsu
See All by Shinya Tsunematsu
GMOペパボでのSREの実践 / SRE Practices of GMO Pepabo, Inc.
tnmt
3
4.8k
ペパボサービスインフラの今までこれから / pepabo infra past and future
tnmt
3
680
知らなかった、時に困るWebサービスのセキュリティ対策 / Where Do We Start With Information Security?
tnmt
19
9.4k
IaaSをいじっている人が PaaSについて考えたこと / Should We Prepare Own PaaS?
tnmt
5
2.2k
成長を支援する “ふりかえり”の技術 / How to lockback using "furik"
tnmt
7
1.7k
こんにちわ福岡 / hello-fukuoka
tnmt
0
1.3k
Inside Nyah & Future - A case of "Private Cloud" using OpenStack -
tnmt
0
270
OpenStackクラスタ間マイグレーション事例 Havana to Mitaka / OpenStack Migration Case (Shift from Havana to Mitaka)
tnmt
1
1.2k
ペパボのプライベートクラウド "Nyah" その後 / Pepabo's PrivateCloud "Nyah" After That
tnmt
8
13k
Other Decks in Technology
See All in Technology
組織観点からIAM Identity CenterとIAMの設計を考える
nrinetcom
PRO
1
170
バイブコーディングと継続的デプロイメント
nwiizo
2
410
それでも私はContextに値を詰めたい | Go Conference 2025 / go conference 2025 fill context
budougumi0617
4
1.2k
PLaMo2シリーズのvLLM実装 / PFN LLM セミナー
pfn
PRO
2
970
BtoBプロダクト開発の深層
16bitidol
0
250
フルカイテン株式会社 エンジニア向け採用資料
fullkaiten
0
9k
神回のメカニズムと再現方法/Mechanisms and Playbook for Kamikai scrumat2025
moriyuya
4
510
From Prompt to Product @ How to Web 2025, Bucharest, Romania
janwerner
0
120
多野優介
tanoyusuke
1
410
Access-what? why and how, A11Y for All - Nordic.js 2025
gdomiciano
1
110
stupid jj tricks
indirect
0
7.9k
10年の共創が示す、これからの開発者と企業の関係 ~ Crossroad
soracom
PRO
1
170
Featured
See All Featured
Testing 201, or: Great Expectations
jmmastey
45
7.7k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
51k
What's in a price? How to price your products and services
michaelherold
246
12k
The Cult of Friendly URLs
andyhume
79
6.6k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Rails Girls Zürich Keynote
gr2m
95
14k
Building Applications with DynamoDB
mza
96
6.6k
Fireside Chat
paigeccino
40
3.7k
How GitHub (no longer) Works
holman
315
140k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
358
30k
Music & Morning Musume
bryan
46
6.8k
Designing Experiences People Love
moore
142
24k
Transcript
2013/11/9 Shinya Tsunematsu @tnmt ͱαʔϏεͷ5ͷาΈ Πϯϑϥվળͷྺ࢙
5ଓ͘αʔϏεͷӡ༻ͬͯ
͘ଓ͘αʔϏεʹ͖ͭ·ͱ͏ ✤ ࢁ͋ΔαʔόؒͰͷඍົͳ༷ࠩ! ✤ ن͕େ͖͍߹ͷมߋͷखؒ! ✤ ΦϯϓϨϛεͷ߹ͷεϖοΫΞοϓʹ͍͍͚ͭͯͳ͍ॊೈੑͷແ ͞! ✤ ϧʔνϯԽ͞Εͨ࡞ۀ͕লྗ͞Ε͍ͯͳ͍͜ͱʹΑΔӡ༻ίετ૿!
✤ Կ͕ى͖͍ͯΔ͔Ѳग़དྷ͍ͯͳ͍αʔϏεϔϧενΣοΫ࣭ͷ ྼԽ
paperboy&co. ϔςϜϧνʔϜ
None
heteml (ϔςϜϧ) ✤ ϩϦϙοϓʂͱฒͿϨϯλϧαʔόͷϥΠϯφοϓ! ✤ 2005ϦϦʔε! ✤ 201310݄ݱࡏαʔό400ऑ! ✤ ৗদ2009͔ΒδϣΠϯ
ٕज़తෛ࠴
ෛ࠴ͱ͍͏ͱωΨςΟϒͳҹʁ ✤ ͔͠͠ඞͣਵ͢ΔͰࣗવͳͷ! ✤ ະવʹ͙ɺൃੜͨ͠߹ʹରॲ͍ͯ͘͠ͷΛ͓Ζ ͔ͦʹ͢ΔͱޙʑࣗͱपΓʹৼΓ͔͔Δ
ΞδΣϯμ ✤ ఏڙ͍ͯ͠ΔαʔϏεͯ͢όʔδϣϯཧԽͰཧ͢Δ! ✤ σϓϩΠαʔόʹର͢ΔมߋπʔϧΛඋɺଐਓԽࡶ͞ Λഉআ! ✤ OSηοτΞοϓΛলྗԽ! ✤ γεςϜߏཧπʔϧΛ༻͍ͯαʔόߏஙͷলྗԽɾ༷Խ
Λߦ͏! ✤ ࢹγεςϜઃఆͷϨϏϡʔɺࢹઃఆͷࣗಈੜ
ͯ͢ΛόʔδϣϯཧԼʹ
ԿΛόʔδϣϯཧ͢Δ͔ ✤ جຊతʹʮͯ͢ʯ
ͱ͍͍͖͑ͳΓશ෦ແཧ ✤ ༏ઌΛ͚ͭΔ! ✤ OSϨΠϠɺηΩϡϦςΟपΓ! ✤ ϛυϧΣΞઃఆ! ✤ cron, ӡ༻εΫϦϓτ!
✤ ࢹιϑτΣΞઃఆ
ΑΓԼͷϨΠϠ͔Βཧ͢Δ ✤ ҙຯͷॏෳΛආ͚ΒΕΔͨΊͦͷ͋ͱͷϝϯςφϯε ੑߴ͘ͳΔ! ✤ ϛυϧΣΞͷηΩϡΞͳઃఆΛܾఆ͢Δ͜ͱ͕༰қ ʹͳΔ! ✤ ҙ֎ͱ͜Ε͕͞Εͯͳ͍έʔε͕ଟ͍
࠷ॳશવཧग़དྷͯͳ͔ͬͨ… ✤ ߏங࣌ظʹΑͬͯઃఆ͕ҧ͏! ✤ ͋Γ·ͤΜ͔ʁ! ✤ iptablesͱtcp_wrapperͰಉ͡ϧʔϧॻ͍ͯΔ! ✤ ϦϓϨΠεΛܦͯɺଘࡏ͍ͯ͠ͳ͍ϗετ͔ ΒͷڐՄઃఆ
όʔδϣϯཧԽʹೖΕ༷Խ ✤ Ұ୴ݱߦಈ͍͍ͯΔαʔόϕʔεͰͱΓ͋͑ͣϦϙ δτϦʹಥͬࠐΉ! ✤ ·͍͠ઃఆͷݟͦ͠ͷޙཧ͢Δ! ✤ มߋτϨʔεՄೳʹͳΔͨΊޙͰݟฦͤΔ
ΦϖϨʔγϣϯπʔϧͷݟ͠
ෳαʔόʹมߋΛՃ͑Δखஈ ✤ όʔδϣϯཧ͕ग़དྷͨΒͦΕΛల։͢Δज़͕ཁΔ! ✤ ͕૿͑Δͱมߋͷख͕ؒେม! ✤ ख͕ؒ૿͑Δͱαʔόؒͷဃ͕૿͑Δѱ॥
sshϧʔϓظ ✤ ຖճଧͭͷ໘ɺઈରϛε͢Δ! ✤ 1ຖ࣮ߦͰγʔέϯγϟϧɺ͕͔͔࣌ؒΔ for host in web{1..100}.heteml.lan;! do!
echo ${host}! ssh ${host} ‘perl -pi -e “s/foo/var/g” /path/to/file’! done
Archerظ ✤ PerlͷσϓϩΠπʔϧ! ✤ YAMLͰఆٛ͞Εͨઃఆʹͱ͖ͮγΣϧίϚϯυ ϑΝΠϧͷσϓϩΠ͕ߦ͑Δ
Archerظ ✤ Pluggable! ✤ Archer::Plugin::RoleLoader! ✤ roles.yaml! ✤ ࢦఆͨ͠ϩʔϧʹ! ✤
ίϚϯυ࣮ߦ! ✤ ϑΝΠϧσϓϩΠ …! ! web:! - web001.heteml.lan! - web002.heteml.lan! mail:! - mail001.heteml.lan! - mail002.heteml.lan! - mail003.heteml.lan! ! …
Archerظ ✤ େྔϗετʹͳΔͱίϚϯυ࣮ߦ݁Ռͷ֬ೝ͕͠ΜͲ ͍! ✤ جຊతʹઃఆϑΝΠϧʹॻ͍ͨ༰Λ࣮ߦ͢ΔͨΊɺ ࡉ͔ͳಈత࡞ۀͷ༻్ʹ͙ͦΘͳ͔ͬͨ! ✤ ಛʹ2ͭΊͷཧ༝͕େมɺಥൃ͢ΔλεΫʹରԠ͖͠ Εͳ͍
ࣗ࡞πʔϧheteloyظ ✤ ༻్ಛԽʢίϚϯυ࣮ߦɺϑΝΠϧσϓϩΠʣ! ✤ ࣗͷPerlπʔϧ! ✤ ʮϔςϜϧʯʹʮσϓϩΠʯ! ✤ σϓϩΠपΓޙड़͢ΔPuppetͱͷੑΛߴΊͨ! ✤
PuppetͷϦϙδτϦͰཧ͞Ε͍ͯΔϑΝΠϧΛ͠ ͘͢ͳ͍ͬͯΔ
ࣗ࡞πʔϧheteloyظ ✤ /var/lib/puppet/data/ ͕ PuppetϦϙδτϦͱͯ͋͠Δ! ✤ /var/lib/puppet/data/modules/web/files/etc/httpd/httpd.conf Λ! ✤ webϩʔϧʢલड़ͷroles.yaml)
ͷϗετʹ͠! ✤ Apache ͷ configtest ͯ͠ OK ͳ߹͚ͩ restart ɺNG ͳ߹ϑΝΠϧϩʔϧόο Ϋ sudo heteloy.pl --role web \! --dist /etc/httpd/httpd.conf \! --mode 644 \! --exec ‘service httpd configtest’ \! --exec ‘service httpd restart’!
ࣗ࡞πʔϧheteloyظ ✤ Α࣮͘ߦ͢ΔλεΫλεΫΛyamlܗࣜʹdump! ✤ Ҏ߱ͦͷyamlΛࢦఆͯ͠ಉ͡ॲཧΛ࣮ߦՄೳ sudo heteloy.pl --role web \!
! --dist /etc/httpd/httpd.conf \! --mode 644 \! --exec ‘service httpd configtest’ \! --exec ‘service httpd restart’ \! --configdump > httpd_task.yaml! sudo heteloy.pl --configloada httpd_task.yaml!
ࣗ࡞πʔϧheteloyظ ✤ ڥґଘΛڐ༰͠൚༻ੑΛࣺͯͨ! ✤ ͷͰެ։ͯ͠ͳ͍Ͱ͢…! ✤ ʑͷӡ༻ͷதͰߦ͏ཻখ͍͕͞εϐʔυΛٻΊ ΒΕΔ࡞ۀʹ͍͍ͯΔ! ✤ PuppetͰͷߏཧ࣠ʹͦΕΛαϙʔτ͢Δҝͷ
πʔϧ͕͋Δͱศར
OSηοτΞοϓࣗಈԽ
৽͍͠ϕϯμͷαʔόΛߪೖ ✤ εϖοΫΞοϓ͕ओ؟! ✤ औΓճ͕͍͢͠͠Α͏IPMIΛࡌ͠ɺԕִʢΦ ϑΟεʣ͔Βͷ࡞ۀ͕ߦ͑ΔΑ͏ʹͨ͠
OSͷΠϯετʔϧDCͤͩͬͨ ✤ IPMI͕͑ΔΑ͏ʹͳΓɺࣗͨͪͰΦϖϨʔγϣϯ Մೳʹ! ✤ ͱ͍͑ɺҰͣͭΠϯετʔϧ͢Δͷେมख͕ؒ ͔͔Δ
Cobbler ✤ http://cobbler.github.com! ✤ ωοτϫʔΫϒʔτͰOSΛΠϯετʔϧ͢Δࡍʹඞཁ ͳίϯϙʔωϯτ͕ͦΖ͍ͬͯΔ! ✤ ݸผͷαʔόͷઃఆใΛதԝͰूதཧՄೳ! ✤ RedHat
Enterprise Linux͓ΑͼͦͷΫϩʔϯOSͰར ༻Մೳ
Cobbler ✤ ૉͷPXEϒʔτͰؤுΕͳ͘ͳ͍͕! ✤ Cobblerdhcpd, pxe, tftp, httpdͳͲΛݸผʹηοτ Ξοϓ͢Δख͕ؒେল͚Δ! ✤
ूதཧ͕༏लɺαʔόઃఆใ (system) ͰIPॏෳ͕ ͋ΔͱΤϥʔʹͳΔͳͲ
CobblerͱPuppetͷॅΈ͚ ✤ CobblerςϯϓϨʔτεχϖοτ͕͑Δ! ✤ ͜ΕΒΛར༻ͯ͠ɺෳϩʔϧ͋ΔαʔόΛΠϯετʔ ϧྃޙଈαʔϏεೖͱ͍ͬͨ͜ͱՄೳͩΖ͏! ✤ kickstartϑΝΠϧϝϯς͠ʹ͍͘! ✤ OSΠϯετʔϧ͑͞ग़དྷΕޙPuppetʹͤΔ
Cobbler ✤ ϗετՃ࣌ɺ͜ͷॲཧޙαʔόΛىಈ͢ΔͱOS ͕Πϯετʔϧ͞ΕΔ ʢωοτϫʔΫ෦ׂѪʣ cobbler system add \! --name
web001.heteml.lan \! --hostname web001.heteml.lan \! --profile CentOS6.4-x86_64 \! --kopts “ksdevide=XX:XX:XX:XX:XX syslog=192.168.X.X” \! --kickstart /var/lib/cobbler/kickstarts/CentOS6-base.ks! ! cobbler sync
γεςϜߏཧπʔϧͷಋೖ
ಋೖͷඞཁੑ ✤ αʔόಋೖϖʔε͕Ճ! ✤ ߏஙͷख͕ؒϘτϧωοΫʹ
طଘαʔϏεͷPuppetಋೖ ✤ खॱॻΛಡΈղ͘! ✤ ඞཁͳ߹खॱॻͷཧ! ✤ ͖ͪΜͱඥղ͖ɺߏཧπʔϧͷઃఆϑΝΠϧʹམͱ ͨ࣌͠ʹͦΕ͕αʔόͷ”༷”ͱͳΔ! ✤ Ұ࣌ظΛࢧ͑ͨൿͷλϨܾͯ͠ѱͷΈͰͳ͍!
✤ ݱ࣮ੈքͷͦΕͱҟͳΔͷ֧፩ʢ͔͘Μʣ͠ͳ͍ҝ
ݕূ͢Δڥ ✤ ࢼͯ͠ഁغͯ͠Γ͕͠ग़དྷΔVMϕʔεͰߦ͏ͷ ͕ΓΓ͍͢! ✤ ϔςϜϧͰϖύϘࣾڥͷKVMཧπʔϧ MaglicaΛͬͯखܰʹVM࡞ɺഁغΛߦͳͬͨ! ✤ ݱࡏͳΒVagrantͳͲ͕͋ΔͨΊͦΕΛར༻͢Δͱ༗ ༻ͦ͏
·ͣ৽نՃϗετ͔Βಋೖ ✤ ৽نϗετطଘαʔόͱͷ͕͠ΒΈ͕ͳ͍! ✤ Ѳ͍ͯ͠ͳ͍༷࿙ΕͳͲ͕͋ΔՄೳੑ͋Γطଘ αʔό৻ॏʹߦ͏ඞཁ͕͋Δ! ✤ طଘϗετͷద༻noop(dry-run)࣮ߦ͠ɺࠩ֬ ೝɺඞཁ͕͋Εमਖ਼ͳͲߦ͍ͭͭॱ࣍ಋೖΛਐΊ ͍ͯΔ
ಋೖ͕ྃͨ͠ޙଓ͘ ✤ ߏཧπʔϧͷهड़ํࣜɺϕετϓϥΫςΟε͕ม ΘΔ͜ͱɺӡ༻͍ͯ͘͠தͰෆཁͳهࡌͳͲग़ͯ ͘Δ! ✤ ࣺͯΔ༐ؾ! ✤ ఆظతʹݟ͠Λ͢Δඞཁ͕͋Δ
ಋೖ͕ྃͨ͠ޙଓ͘ ✤ ඞཁͳઃఆόʔδϣϯཧ͞Ε͍ͯΔ! ✤ ༷ͷมߋݟ͠ઃఆϑΝΠϧϕʔεͰٞ͢Δ! ✤ ϖύϘͰGH:EΛར༻͍ͯ͠ΔͨΊɺPull Requestϕʔ εͰ্هΛߦͳ͍ͬͯΔ! ✤
ϨϏϡʔͯ͠Β͑Δͱਫ਼͕͕͋Δ! ✤ ͋ͱ҆৺͢Δ
ࢹγεςϜͷӡ༻վળ
ࢹ͕ἧͬͯͳ͍ ✤ ࢹ߲͕ࡏΔͷͱແ͍ͷ͕͋Δ! ✤ ϗετΛՃ͢Δ͝ͱʹखՃ! ✤ ͦͦࢹ͞Ε͍ͯͳ͍ϗετ…ʁ
αʔόͷҰཡ࣋ͬͯΔ ✤ લड़ͷroles.yaml! ✤ ͜͜ʹཧ͍ͯ͠Δαʔόཏ͞Ε͍ͯΔ
ࢹઃఆͷࣗಈੜ ✤ Nagios! ✤ munin! ✤ roles.yaml Λͱʹ conf Λࣗಈੜ͢Δ!
✤ શαʔόڞ௨ͷࢹ (σΟεΫI/O, L/A)! ✤ ϩʔϧݻ༗ͷࢹ (webͩͬͨΒHTTPૄ௨)! ✤ ϗετݻ༗ͷઃఆ (ಛఆαʔόαʔό্Ͱύϥϝʔλ্ॻ͖ʣ
มߋ࣌ ✤ طଘͷࢹγεςϜʢखಈՃʣͱ৽ࢹγεςϜʢઃ ఆࣗಈੜʣΛฒߦͯ͠Քಇͤ͞Δ! ✤ Nagios৽ࢹγεςϜΞϥʔτඈ͞ͳ͍! ✤ ेʹ֬ೝ͔ͯ͠Β͍͑ͰΓସ͑
ৼΓฦͬͯ
ྺ࢙͕͍ͷͰτϨϯυมΘΔ ✤ Puppetmaster agentܕͰͬͯΔ͚Ͳɺ࠷ۙελ ϯυΞϩϯܕ͕ྲྀߦͬͯΔʢςετۦಈΠϯϑϥߏஙʣ ✤ SerfͳͲʹΑΔࣗతʢಈతʣͳϊʔυՃɾઃఆ ՃͳͲ! ✤ ͳͲͳͲ
Δ͔͠ͳ͍ ✤ WebͷαʔϏεͳݶΓ͕ͳ͍ɺӡ༻ʹऴΘΓ͕ ͳ͍! ✤ ͚Ͳޮతʹɺָ͘͢͠Δ͜ͱग़དྷΔ! ✤ Ұ൪ϘτϧωοΫʢࡶɺ͕͔͔࣌ؒΔʣ࡞ۀ͔Βૣ ΊʹऔΓΉ! ✤
ಓʹΔ
ྫ ✤ ࠷ۙϝϯς͕ଟ͍! ✤ ରϗετ͕ҧ͏͚ͩͰखॱ͍ಓ͕ฉ͘! ✤ खॱॻੜʢςϯϓϨʔτԽʣ͠Α͏ʂ
खॱΛςϯϓϨʔτԽ export OLDHOST=<%= @config['src1_hostname'] %>! export NEWHOST=<%= @config['dst_lan_ip'] %>! rsync
-rlptDv /data/backup/heteml/$OLDHOST/path/from/ $NEWHOST::data_to/ src1_hostname : web001! src1_lan_ip : 192.168.1.10! ! dst_hostname : web002! dst_lan_ip : 192.168.1.11!
खॱΛςϯϓϨʔτԽ genecon -t README.md -c config.yaml > web001maintenance.md export OLDHOST=web001!
export NEWHOST=192.168.1.11! rsync -rlptDv /data/backup/heteml/$OLDHOST/path/from/ $NEWHOST::data_to/ ✤ genecon https://github.com/glidenote/genecon
͋Δఔͷڳඞཁ ✤ ৽͍͠ͷͷಋೖԿ͔ͷ৽༐ؾ͕ཁΔ! ✤ ͰΔ࣌ʹΒͳ͍ͱ͍͚ͳ͍! ✤ ཪ͚Ͱ͋Δ֬ೝςετͳͲͰཪ͚Λͯ҆͠৺ ͢Δ
ࠓճͷ༰ɺ͞Βʹৄ͍͠༰ ✤ WEB+DB PRESS Vol.75! ✤ ʙෳࡶੑͷ૿େͱڥͷมԽʹରԠ͢Δʙ ܧଓతWebαʔϏεվ ળΨΠυ! ✤
ΞϓϦɺνʔϜϏϧσΟϯάͷվળͳͲଞͷτϐοΫ߹Θͤͯࡌͬ ͯ·͢
Ұॹʹಇ͘ϝϯόʔΛืू͍ͯ͠· ͢ ✤ http://www.paperboy.co.jp/recruit/engineer_career/! ✤ ϔςϜϧνʔϜͷΠϯϑϥΤϯδχΞืूதͰ͢
͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠