インフラチームを持たない会社でのインフラ運用

Slide 1

Slide 1 text

ΠϯϑϥνʔϜ͕ແ͍ձࣾ ͰͷΠϯϑϥӡ༻ 13೥7݄20೔౔༵೔

Slide 2

Slide 2 text

࢝ΊΔDevOps ΞΧπΩͰͷDevOpsࣄྫ ͲͷΑ͏ʹ࢝ΊΕ͹͍͍ͷ͔ 13೥7݄20೔౔༵೔

Slide 3

Slide 3 text

ࣗݾ঺հ ాதɹ༐ี @csouls ϋοΧʔ Lv.3ɻϢʔβܥSIerͰERPͷӡ༻΍Π ϯϑϥߏஙͷۀ຿Λܦݧͨ͠ޙɺݱ৬΁స৬ɻ RubyͱΠϯϑϥ͕޷෺ͰɺΞϓϦ։ൃͱฒߦ ͯ͠ΞΧπΩͷΠϯϑϥΛӡ༻͍ͯ͠Δ 13೥7݄20೔౔༵೔

Slide 4

Slide 4 text

ΞΧπΩ ιʔγϟϧήʔϜ࡞͍ͬͯΔձࣾ ૑ۀ͔࣌Β࢖͍ͬͯΔ΋ͷ͸ Ruby on Rails ͱ Nginx ΤϯδχΞ 9 ਓɻ։ൃͭͭ͠Πϯϑϥӡ༻ ӡ༻ͷޮ཰Խ͕ॏཁ 13೥7݄20೔౔༵೔

Slide 5

Slide 5 text

Πϯϑϥͷྺ࢙ ૑ۀظ 2010~ ߏஙͷࣗಈԽ 2011~ ӡ༻҆ఆԽ 2012~ ӡ༻ࣗಈԽ 2013~ 13೥7݄20೔౔༵೔

Slide 6

Slide 6 text

૑ۀظ(҉ࠇ࣌୅) 2010~ ผͷΫϥ΢υαʔϏε͔ΒAWSʹ৐Γ׵ ͑ Load Balancer ͷੑೳͱΠϯελϯε࡞ ੒ͷख͔ܰ͞ΒɺAWS΁ ࠷ॳ͸ख࡞ۀͰαʔόߏஙɻצ͕ॏཁ 13೥7݄20೔౔༵೔

Slide 7

Slide 7 text

૑ۀظ(҉ࠇ࣌୅) 2010~ 6~7 ສDAU Session ͷอଘઌ͕ MySQL DB ʹ → Master DBʹॲཧ͕ूத 5෼ஔ͖ʹམͪΔDBΛ࠶ىಈ͢Δ࡞ۀɻ ప໷Ͱ memcached Λಋೖ͠ɺͳΜͱ͔ མͪͳ͍Α͏ʹͳͬͨ͜ͱ΋ɻ 13೥7݄20೔౔༵೔

Slide 8

Slide 8 text

Ϣʔβ→ ←MySQL 13೥7݄20೔౔༵೔

Slide 9

Slide 9 text

ߏஙͷࣗಈԽ 2011~ αʔόઃఆεΫϦϓτ(shell)Λ Git Ͱ؅ཧ AutoScaling + εϙοτΠϯελϯεͰӡ ༻͍͕ͯͨ͠ɺࢥͬͨΑΓίετ͕཈͑Β Εͣɺαʔό΋҆ఆ͠ͳ͔ͬͨͷͰࢭΊͨ εϙοτΠϯελϯεྲྀߦΓͩͨ͠ͷ΋ɺ ҆ఆ͠ͳ͔ͬͨݪҼ 13೥7݄20೔౔༵೔

Slide 10

Slide 10 text

ӡ༻҆ఆԽ 2012~ Auto Scaling πʔϧΛ࡞੒ɻӡ༻͕҆ఆ͠ ͯਓؒΒ͍͠ੜ׆͕ग़དྷΔΑ͏ʹͳͬͨ 13೥7݄20೔౔༵೔

Slide 11

Slide 11 text

ӡ༻҆ఆԽ 2012~ EC2ͷλάʹΑͬͯ໾ׂΛ؅ཧ EC2ʹ "Role" λάΛ෇͚ͯɺ໾ׂΛࢦఆ /etc/rc.local Ͱ git pull & λάʹԠͨ͡ε ΫϦϓτΛ࣮ߦ ShellͰႈ౳ੑΛอͭͨΊʹؤு͍ͬͯͨ 13೥7݄20೔౔༵೔

Slide 12

Slide 12 text

NFS 13೥7݄20೔౔༵೔

Slide 13

Slide 13 text

ӡ༻҆ఆԽ 2012~ NFS αʔό͸SPOF كʹ Buffer cache ͕ංେԽͯ͠ো֐ʹͳΔ 13೥7݄20೔౔༵೔

Slide 14

Slide 14 text

ӡ༻ࣗಈԽ 2013~ Chef, Capistrano ಋೖ NFSΛഇࢭͯ͠ɺSPOF͕ແ͘ͳͬͨ 13೥7݄20೔౔༵೔

Slide 15

Slide 15 text

13೥7݄20೔౔༵೔

Slide 16

Slide 16 text

πʔϧ ෺ཧ OSઃఆɾϛυϧ ΢ΣΞ ΞϓϦέʔγϣϯ AWS Chef Capistrano 13೥7݄20೔౔༵೔

Slide 17

Slide 17 text

ϛυϧ΢ΣΞઃఆ Chef Solo knife-solo Berkshelf : CookbookΛ؅ཧ serverspec / Vagrant : ςετ 13೥7݄20೔౔༵೔

Slide 18

Slide 18 text

ϛυϧ΢ΣΞઃఆ Chef Solo αʔυύʔςΟ cookbook Λ Berkshelf ͷ ֎Ͱ؅ཧͨ͠Γɺfork ͢Δͷ͸Ξϯνύ λʔϯ ΞΧπΩͰ͸ɺsite-cookbooksͰ্ॻ͖͠ ͍ͯΔ 13೥7݄20೔౔༵೔

Slide 19

Slide 19 text

ϛυϧ΢ΣΞઃఆ Chef Solo ൓ө͸खಈɻ·ͩ production ؀ڥ΁ͷࣗ ಈ൓ө͸ͪΐͬͱා͍... serverspec ͕ॆ࣮ͨ͠Β Git ϦϙδτϦ ͷ master ϒϥϯνͷߋ৽ʹԠͯࣗ͡ಈ Ͱద༻͢Δ 13೥7݄20೔౔༵೔

Slide 20

Slide 20 text

σϓϩΠ Capistrano EC2ͷRoleλά͔ΒɺσϓϩΠΛ࣮ߦ͢ ΔΞϓϦέʔγϣϯαʔόΛऔಘ 13೥7݄20೔౔༵೔

Slide 21

Slide 21 text

# config/deploy.rb def tagged_servers(tag_key, tag_value, default=[]) @ec2 ||= AWS::EC2.new(ec2_endpoint: 'ec2.ap-‐northeast-‐1.amazonaws.com') ret = @ec2.instances.map do |instance| next if instance.tags[tag_key] != tag_value next if instance.status != :running instance.dns_name.empty? ? instance.ip_address : instance.dns_name end.compact return default if ret.empty? ret end def tag(tag_value, *args) AWS.memoize { tagged_servers(tag_key, tag_value).each do |host| server(host, *args) end } end # config/deploy/environment.rb tag 'app', :web, :app σϓϩΠ 13೥7݄20೔౔༵೔

Slide 22

Slide 22 text

σϓϩΠ AWS ͷ λά͸γϯϓϧ͚ͩͲɺ௒ศར Infrastructure as Code Λࢧ͑Δλά 13೥7݄20೔౔༵೔

Slide 23

Slide 23 text

؂ࢹ ؂ࢹϨΠϠ πʔϧ ର৅ OS؂ࢹ Amazon CloudWatch CPU / ϝϞϦ / σΟεΫ / Πϯελϯε਺ ϓϩηε؂ࢹ God Unicorn / Resque ϛυϧ΢ΣΞ؂ࢹ nagios MySQL Slow query౳ ΞϓϦέʔγϣϯ؂ࢹ NewRelic ύϑΥʔϚϯε 13೥7݄20೔౔༵೔

Slide 24

Slide 24 text

God ϓϩηε؂ࢹπʔϧ ϝϞϦΛ࢖͍͗ͨ࣌͢ɺ๫૸ͨ࣌͠ͷอ ݥ monit Ͱ͸ͳ͘ God Λ࢖͍ͬͯΔཧ༝͸ Ruby Ͱॻ͚Δ͔Β 13೥7݄20೔౔༵೔

Slide 25

Slide 25 text

God Ϛελʔϓϩηε͸GodͰ؂ࢹ͠ɺWorker͸ kzk/unicorn-worker-killer Ͱɺఆظతʹ࠶ىಈ require ::File.expand_path('../config/environment', __FILE__) require 'unicorn/oob_gc' require 'unicorn/worker_killer' # ϦΫΤετΛ࣮ߦ͍ͯ͠Δͱ͖͸GC͠ͳ͍ use Unicorn::OobGC # 3072~4096ϦΫΤετ࣮ߦͨ͠Β࠶ىಈ͢Δ use Unicorn::WorkerKiller::MaxRequests, 3072, 4096 run XXXXXXX::Application 13೥7݄20೔౔༵೔

Slide 26

Slide 26 text

NewRelic νϡʔχϯά͢΂͖ͱ͜Ζ͕Ұ໨Ͱ෼͔Δ DOM processing ͕ࢧ഑తͳͷͰɺCSS / JavaScriptΛνϡʔχϯά͢Δ ͱɺύϑΥʔϚϯε͕վળ͢Δ Applicationαʔόͷ࣮ߦ࣌ؒ͸Ԡ౴࣌ؒͷ 1 / 80 ͘Β͍ 13೥7݄20೔౔༵೔

Slide 27

Slide 27 text

NewRelic AppServer ͷৄࡉ΍ɺϖʔδৄࡉ͕ݟΕΔ 13೥7݄20೔౔༵೔

Slide 28

Slide 28 text

Ruby on Rails͚ͩ͡Όͳ͍ NewRelic 13೥7݄20೔౔༵೔

Slide 29

Slide 29 text

ϓϥάΠϯ͕ॆ࣮͖ͯͨ͠ ಋೖϓϥάΠϯ : EBS,EC2,ELB,MySQL,Newvem,Nginx,R DS,Redis NewRelic 13೥7݄20೔౔༵೔

Slide 30

Slide 30 text

ΞϓϦέʔγϣϯαʔόͷࢮ׆؂ࢹʹ΋࢖ ͍ͬͯΔ NewRelic 13೥7݄20೔౔༵೔

Slide 31

Slide 31 text

ӡ༻ AutoScaling 13೥7݄20೔౔༵೔

Slide 32

Slide 32 text

Auto Scaling ࣦഊ Scaling Policy ͷઃఆϛεɻᮢ஋पลͷෛ ՙʹΑΓɺEC2͕සൟʹ࠶ىಈ͢Δ → ىಈ͢Δͱ1h ෼ͷඅ༻͕͔͔Δͷ Ͱɺαʔόඅ༻͕Ұ࣌2ഒʹ 13೥7݄20೔౔༵೔

Slide 33

Slide 33 text

ෛՙͷ܏޲͸PVʹै ͏ CPUෛՙΑΓ΋PVͷ ํ͕҆ఆͨ͠ۂઢΛ ඳ͘ Auto Scaling 13೥7݄20೔౔༵೔

Slide 34

Slide 34 text

Auto Scaling PV਺ΛݩʹScaleͯ͠΄͍͠ ᮢ஋पลͰසൟʹىಈఀࢭΛ܁Γฦͨ͘͠ ͳ͍ ։ൃऀ͕ࣦഊͤͣ؆୯ʹ࢖͑Δ 13೥7݄20೔౔༵೔

Slide 35

Slide 35 text

Auto Scaling Chariot : αʔό1୆ͰॲཧͰ͖ΔPV਺Λݩ ʹεέʔϧ samplingͷͲΕ͔1ͭͰ΋(PV਺/base)͕ՔಇΠϯελϯε਺Λ௒͍͑ͯ ͨΒɺΠϯελϯε਺Λ(ฏۉPV/base)ʹ߹ΘͤΔ samplingͷશͯͷPV਺Ͱ(PV਺/base)͕ݱࡏͷՔಇΠϯελϯε਺ΛԼ ճ͍͍ͬͯΕ͹ɺΠϯελϯε਺Λ(ฏۉPV/base)ʹ߹ΘͤΔ ͦΕҎ֎͸Կ΋͠ͳ͍ 13೥7݄20೔౔༵೔

Slide 36

Slide 36 text

Chariot $ ruby bin/watcher app-‐name 10 minuts PV dataset: 2013-‐07-‐19 17:06:00 +0900: 1416.0 2013-‐07-‐19 17:07:00 +0900: 1269.0 2013-‐07-‐19 17:08:00 +0900: 1220.0 2013-‐07-‐19 17:09:00 +0900: 1286.0 2013-‐07-‐19 17:10:00 +0900: 1293.0 2013-‐07-‐19 17:11:00 +0900: 1352.0 2013-‐07-‐19 17:12:00 +0900: 1252.0 2013-‐07-‐19 17:13:00 +0900: 1248.0 2013-‐07-‐19 17:14:00 +0900: 1232.0 2013-‐07-‐19 17:15:00 +0900: 1266.0 10 minuts PV average: 1283.4 Current [3] Expect [2] but config min value is [3] -‐-‐-‐ Do nothing -‐-‐-‐ # AWSͷઃఆ access_key_id: AWS_ACCESS_KEY_ID secret_access_key: AWS_SECRET_KEY ec2_endpoint: ec2.ap-‐northeast-‐1.amazonaws.com cloud_watch_endpoint: monitoring.ap-‐northeast-‐1.amazonaws.com elb_endpoint: elasticloadbalancing.ap-‐northeast-‐1.amazonaws.com # ΞϓϦͷઃఆ min: 3 # ࠷௿ىಈΠϯελϯε਺ event-‐min: 20130722_2220-‐20130722_2310: 15 20130723_2115-‐20130723_2205: 30 20130723_2205-‐20130723_2255: 15 sampling: 10 # CloudWatch͔Βऔಘ͢ΔαϯϓϦϯά਺(1෼ʹ͖ͭ1ͭ) base: 1000 # 1Πϯελϯε͕ॲཧͰ͖Δ෼ؒPV਺ 13೥7݄20೔౔༵೔

Slide 37

Slide 37 text

҆ఆͯ͠ӡ༻Ͱ͖ΔΑ͏ʹͳͬͨ ӡ༻ίετ͕ࢥ͍௨Γʹ࡟ݮͰ͖ͨ ·ͩެ։ͯ͠ͳ͍ Chariot 13೥7݄20೔౔༵೔

Slide 38

Slide 38 text

ίϥϘϨʔγϣϯ Github Pull Request, Issue Redmine DevOpsλεΫɺΠϯϑϥWiki HipChat 13೥7݄20೔౔༵೔

Slide 39

Slide 39 text

Cookbook ΍ Chariot ͷ໰୊͸ Github ͷ Issue ʹ ొ࿥ͯ͠ɺमਖ਼ग़དྷͦ͏ͳਓ͕ Pull Request ͢Δ ͜Ε͔Βಋೖ͍ͨ͠ࣄ΍ӡ༻ͷ໰୊͸ɺ Redmine ʹνέοτొ࿥͢Δ ίϥϘϨʔγϣϯ 13೥7݄20೔౔༵೔

Slide 40

Slide 40 text

ίϥϘϨʔγϣϯ HipChat νϟοτπʔϧ Campﬁre ͷ Atlassian൛ ػೳ͸΄΅Ұॹ͚ͩͲɺMac ωΠςΟϒΞ ϓϦ͕࢖͍΍͍͢ 13೥7݄20೔౔༵೔

Slide 41

Slide 41 text

13೥7݄20೔౔༵೔

Slide 42

Slide 42 text

ίϥϘϨʔγϣϯ Github࿈ܞ͕஍ຯʹศར 13೥7݄20೔౔༵೔

Slide 43

Slide 43 text

ίϥϘϨʔγϣϯ ΄ͷ΅ͷͱͨ͠งғؾ 13೥7݄20೔౔༵೔

Slide 44

Slide 44 text

໨ࢦ͍ͯ͠Δ΋ͷ : HipChat ͱ Hubot Ͱ ൃ ݴΛरͬͯσϓϩΠ ΤϯδχΞ͚ͩͰ͸ͳ͘ɺσβΠφʔ΍ σʔλΛߋ৽͢ΔσΟϨΫλʔ͕Χδϡ ΞϧʹσϓϩΠग़དྷΔΑ͏ʹ͍ͨ͠ ίϥϘϨʔγϣϯ 13೥7݄20೔౔༵೔

Slide 45

Slide 45 text

DevOpsΛਪਐ͢Δ ૊৫͕খ͘͞ɺ։ൃ෦໳ͱӡ༻෦໳͕෼͔ Ε͍ͯͳ͍ͷͰɺDev ͱ Ops ͷ͕֞ࠜͳ ͍ DevOpsΛ࢝ΊΔো֐͸গͳ͍͚Ͳɺจ Խ͸ࢀߟʹͳΔ͸ͣ 13೥7݄20೔౔༵೔

Slide 46

Slide 46 text

จԽ ΤϯδχΞνʔϜͷཧ೦ 2. Keep changing มԽͷத৺ʹ਎Λஔ͖ɺৗʹվળ͠ଓ͚Δ "ٕज़ͷྲྀΕʹ෇͍͍͚ͯͩ͘Ͱ͸ͳ͘ɺྲྀΕͷதʹੵۃతʹ਎Λஔ͖ɺৗ ʹࣗ෼ࣗ਎͕มԽͷத৺఺ʹ͍ΔΑ͏ʹมΘΓଓ͚Δɻ" 5. Do it yourself ࣗΒͷ؀ڥ͸ࣗΒ૑Γͩ͠ɺվળ͢Δ "؀ڥ͸୭͔͕༩͑ͯ͘ΕΔ΋ͷͰ΋ɺ࠷ॳ͔Βͦ͜ʹ͋Δ΋ͷͰ΋ͳ͍ɻ ࣗΒ͕׆ಈ͢ΔͨΊʹ࠷దͳ؀ڥ͸ࣗΒ͕ߟ͑ɺࣗΒखΛಈ͔͠૑Γͩ࢟͢ ੎ͰऔΓ૊Ήɻ" 13೥7݄20೔౔༵೔

Slide 47

Slide 47 text

શࣾ࠷దͰ͸ͳ͘ɺνʔϜ࠷ద ϓϩμΫτ͝ͱͷνʔϜ ϓϩμΫτΛ࡞ΔͨΊʹ࠷దͳٕज़Λɺ طଘͷࢿ࢈ΛؚΊɺݸʑͷνʔϜ͕ݕ౼ ͢Δ จԽ 13೥7݄20೔౔༵೔

Slide 48

Slide 48 text

ڞ௨ݴޠΛ͍࣋ͬͯΔ ͱΓ͋͑ͣ Ruby Ͱॻ͚͹ΈΜͳ෼͔ Δͱ͍͏ͷ͸ɺେ͖͍ จԽ 13೥7݄20೔౔༵೔

Slide 49

Slide 49 text

ͲͷΑ͏ʹͯ͠޿Ί͍͔ͯ͘ খ࢝͘͞ΊΔ ·ͣ͸ Chef Solo ͔Β ৽نPJͷαʔόߏஙεΫϦϓτΛ Chef ʹஔ ͖׵͑ͯΈΔͱ͜Ζ͔Β ੒ޭମݧΛڞ༗͢Δ πʔϧΛཧղͯ͠΋͏ͷ͸೉͍͚͠Ͳɺετ ʔϦʔΛཧղͯ͠΋Β͏ͷ͸؆୯ 13೥7݄20೔౔༵೔