Consul in Production

Transcript

1. DPOTVMͷ༻్͍Ζ͍Ζ
   ࣲాതࢤ(.01&1"#0JOD
   $POTVM$BTVBM5BMLT
   $POTVMJO1SPEVDUJPO
   

   View Slide

2. νʔϑΤϯδχΞ
   ࣲాതࢤ!ITCU
   ٕज़෦ٕज़ج൫νʔϜ
   IUUQTXXXITCUPSH
   

   View Slide

3. self.introduce
   =>
   {
   github: “@hsbt”,
   commit_bits: [
   “ruby”, “rake”, “rubygems”, “rdoc”,
   “psych”, “syck”, “ruby-build”, “jenkins.rb”…
   ],
   sites: [
   “hsbt.org”, “ruby-lang.org”, “rubyci.org”,
   “railsgirls.com”, “railsgirls.jp”
   ],
   }
   

   View Slide

4. DPOTVMͰ΍ͬͯΔ͜ͱΛ঺հ
   wNJOOF NJOOFDPN
   Ͱ΍ͬͯΔ͜ͱJO
   wNJOOF͸0QFO4UBDLׂɺ"84ׂͷNVMUJ%$ɺ7.͸͘Β͍ɺશ
   ෦ʹDPOTVMೖΕͯ·͢
   wDPOTVMͷED͸࢖ͬͯ·ͤΜɺ0QFO4UBDLͱ"84ͰผͷΫϥελ
   wDPOTVMEOTNBTRͰ΍Δ಺෦%/4͸লུ
   wTUSFUDIFSΛ࢖ͬͨσϓϩΠ͸লུ
   wDPOTVMͰUMTTFTTJPOUJDLFUͷߋ৽͸লུ
   

   View Slide

5. ͓·͚systemd
   [Unit]
   Description=Consul is a tool for service discovery and configuration. Consul is distributed, highly available, and extremely scal
   Documentation=http://www.consul.io
   After=network-online.target
   Wants=network-online.target
   [Service]
   User=consul
   Group=consul
   EnvironmentFile=-/etc/sysconfig/consul
   ExecStart=/usr/bin/consul $CMD_OPTS
   ExecReload=/bin/kill -HUP $MAINPID
   KillSignal=SIGINT
   Restart=on-failure
   LimitNOFILE=8192
   [Install]
   WantedBy=multi-user.target
   

   View Slide

6. /FX3FMJDͷϗετબఆ
   

   View Slide

7. ͜͏͍͏͜ͱΛ΍Γ͍ͨ
   wNJOOFͰ͸ΞϓϦέʔγϣϯͷύϑΥʔϚϯεϞχλϦϯάʹ/FX3FMJD
   IUUQTOFXSFMJDDPN
   Λ࢖͍ͬͯΔ
   w/FX3FMJD͸ߴ͍ IPTU
   
   ˠόϧΫσΟεΧ΢ϯτΛަব͢ΔͱIPTU͘Β͍·ͰԼ͕Δ
   wͦΕͰ΋αʔϏεͷϩʔϧ͕ͭ͋Ε͹࠷௿IPTUʹೖΕΔඞཁ͕͋Δ
   wϩʔϧͷத͔Β͍͍ײ͡ʹ୆͚ͩબΜͰ/FX3FMJDΛ༗ޮʹ͍ͨ͠
   

   View Slide

8. ͜͏΍ͬͨ
   w/FX3FMJDͷઃఆϑΝΠϧ 3BJMT༻
   ʹDPOTVM͔ΒϗετΛऔಘͯ͠ϗε
   τ͚ͩ༗ޮͱ͢ΔΑ͏ͳίʔυΛຒΊࠐΉ
   wOFXSFMJD@SQNͷHFN͕ϩʔυ͞Εͨ࣌ʹݺͼग़͞Εͯཉ͍͠ͷͰ3VCZͷ
   ίʔυ͸A3BJMTSPPUKPJO bMJC`
   Aʹஔ͘
   w$POTVM΁ͷΞΫηε͸%JQMPNBU IUUQTHJUIVCDPN8F"SF'BSN(FFL
   EJQMPNBU
   Λ࢖͏
   wΦϦδφϧIUUQTNBLBOESBDBSETDPNNBLBOESBFOBCMF
   OFXSFMJDNPOJUPSJOHGPSSBJMTPOTQFDJpDIPTUTPOMZ
   

   View Slide

9. ͜Μͳײ͡
   common: &default_settings
   (snip)
   test:
   <<: *default_settings
   monitor_mode: false
   production:
   <<: *default_settings
   monitor_mode: <%= CustomNewRelicConfiguration.enable_on_current_host? rescue false %>
   

   View Slide

10. ͜Μͳײ͡
    
    require 'socket'
    class CustomNewRelicConfiguration
    def self.enable_on_current_host?
    current_hostname = Socket.gethostname.strip
    Rails.logger.info ">>> Checking whether to enable NewRelic on host '#{current_hostname}'"
    if newrelic_hosts == current_hostname
    Rails.logger.info ">>> NewRelic should be enabled on this host."
    true
    else
    Rails.logger.info ">>> NewRelic is disabled on this host! Edit config/newrelic.yml to enable."
    false
    end
    end
    def self.newrelic_hosts
    require 'diplomat'
    Diplomat.configuration.url = "http://127.0.0.1:8500"
    node = Diplomat::Node.get_all.
    select{|n| n.Node =~ /^www-/ }.
    sort_by{|n| n.Node }.
    first.Node
    "#{node}.minne.com"
    end
    end
    

    View Slide

11. TJEFLJRTDIFEVMFS
    

    View Slide

12. ͜͏͍͏͜ͱΛ΍Γ͍ͨ
    wDSPO͸WN͕410'ͱͳΓ͕ͪͳͷͰ΍Ί͍ͨ
    wIUUQTHJUIVCDPNNPPWFJUTJEFLJRTDIFEVMFSΛ࢖͏ͱTJEFLJRʹ
    QFSJPEJDBMKPC͕࢖͑ΔΑ͏ʹͳΔ TJEFLJRQSPͰ΋Ͱ͖Δ
    
    wTJEFLJRTDIFEVMFS͸୯७ʹ࢖͏ͱXPSLFSશ୆Ͱ༗ޮʹͳͬͯɺδϣϒ͕ॏ
    ෳͯ͠ࠔΔ
    wDPOTVMͰৗʹҰ୆͚ͩTJEFLJRTDIFEVMFSΛ༗ޮʹ͍ͨ͠
    

    View Slide

13. ֓೦ਤ
    sidekiq
    worker
    sidekiq
    worker
    &
    scheduler
    redis
    sidekiq
    worker
    sidekiq
    worker
    sidekiq
    worker
    sidekiq
    worker
    sidekiq
    worker
    redis
    sidekiq
    worker
    sidekiq
    worker
    sidekiq
    worker
    

    View Slide

14. ͜͏΍ͬͨ
    wDPOTVMΛ࢖ͬͯɺҰఆͷϧʔϧͰҰ୆͚ͩTDIFEVMFSΛ༗ޮͱ͢ΔΑ͏ͳϗ
    ετωʔϜΛऔΓग़͢
    wTDIFEVMFS͕͍ͳ͘ͳΔͱࠔΔͷͰɺTDIFEVMFSϗετ͕ϓϥοτϑΥʔϜͷ
    τϥϒϧͰམͪͨ৔߹͸ࣗಈͰ࠶બग़͢Δ ·ͩͰ͖ͯͳ͍ʜ
    
    

    View Slide

15. ͜Μͳײ͡
    Sidekiq.configure_server do |config|
    if Rails.env.production? # for sidekiq-scheduler
    config.on(:startup) do
    if Sidekiq::Scheduler.master_server?
    Sidekiq::Scheduler.register_server
    Sidekiq::Scheduler.reload_schedule!
    else
    Sidekiq::Scheduler.enabled = false
    end
    end
    %i(quiet shutdown).each do |state|
    config.on(state) do
    Sidekiq::Scheduler.reset_master_server!
    end
    end
    end
    end
    module Sidekiq
    class Scheduler
    def self.master_server
    Diplomat::Node.get_all.
    select{|n| n.Node =~ /^worker-/ }.
    sort_by{|n| n.Node }.
    first.Node
    end
    def self.master_server?
    Socket.gethostname.chomp == self.master_server
    end
    def self.register_server
    Sidekiq.redis {|c| c.set(‘sidekiq:schedules:master’, Socket.gethostname.chomp) }
    end
    def self.reset_master_server!
    if Socket.gethostname.chomp == Sidekiq.redis {|c| c.get('sidekiq:schedules:master')}
    Sidekiq.redis {|conn| conn.del('sidekiq:schedules:master')}
    end
    end
    end
    end
    

    View Slide

16. ϩʔυόϥϯαͷ%/4ߋ৽
    

    View Slide

17. ͜͏͍͏͜ͱΛ΍Γ͍ͨ
    wMPBECBMBODFSͱͯ͠OHJOYΛO୆ར༻͍ͯ͠Δ
    wOHJOYͷVQTUSFBNͱͯ͠N୆഑ஔ͠ɺΫϩεϝογϡͰܨ͍Ͱ͍Δ
    wMPBECBMBODFSΛ%/4ϥ΢ϯυϩϏϯͰ෼ࢄΞΫηε͍ͯ͠Δ
    wMPBECBMBODFS͕ϐϤͬͨ࣌ʹ%/4ϥ΢ϯυϩϏϯͷର৅͔ΒࣗಈͰ֎ͨ͠
    ͍
    

    View Slide

18. ֓೦ਤ
    application
    server
    DNS
    (Route53)
    load
    balancer
    load
    balancer
    application
    server
    application
    server
    ো֐ൃੜ
    application
    server
    load
    balancer
    load
    balancer
    application
    server
    application
    server
    ࣗಈִ཭
    DNS
    (Route53)
    

    View Slide

19. ͜͏΍ͬͨ
    wIUUQTHJUIVCDPNVE[VSBDPMMFDUPSΛར༻
    wDPOTVMͷTFSWJDFঢ়گΛ؂ࢹ OBHJPTQMVHJOBMM࢖ͬͯ·͕ͨ͠NBDLFSFM
    ʹҠߦத
    
    wDPOTVMͷTFSWJDF MPBECBMBODFSͷOHJOY
    ͕μ΢ϯ͍ͯ͠Δ৔߹͸֘౰ͷ
    DPOTVMOPEFΛDPOTVMXBUDIͱDPMMFDUPSΛ༻͍ͯEOT͔Β੾Γ͸ͣ͠
    wৄࡉ͸ઃఆϑΝΠϧͳͲ͸3&"%.&NEΛΈ͍ͯͩ͘͞ লུ
    
    

    View Slide

20. ͜Μͳײ͡
    

    View Slide

21. OHJOYͷVQTUSFBNͷࣗಈߋ৽
    

    View Slide

22. ͜͏͍͏͜ͱΛ΍Γ͍ͨ
    wMPBECBMBODFSͰ࢖͍ͬͯΔOHJOYͷVQTUSFBNΛࣗಈߋ৽͍ͨ͠
    wIUUQTHJUIVCDPNIBTIJDPSQDPOTVMUFNQMBUFͰOHJOYͷ
    DPOpHVSBUJPOΛࣗಈੜ੒
    wҟৗ͕͋ΔWN͸VQTUSFBN͔Β֎͢͠ɺ৽্ཱͪ͛ͨ͘͠WN͸ࣗಈͰ
    MPBECBMBODFSʹKPJO͢Δ
    w஍ཕIUUQUPNPIJTBPEBDPNQPTUTVTF@TZTUFNE@XJUI@DPOTVM
    UFNQMBUFIUNM
    

    View Slide

23. ֓೦ਤ
    application
    server
    DNS
    (Route53)
    load
    balancer
    load
    balancer
    application
    server
    application
    server
    ো֐ൃੜ
    application
    server
    DNS
    (Route53)
    load
    balancer
    load
    balancer
    application
    server
    application
    server
    ࣗಈִ཭
    

    View Slide

24. ͜͏΍ͬͨ
    wMPBECBMBODFSʹDPOTVMUFNQMBUFΛೖΕͯૉ๿ʹ࣮ߦ
    wDPOpHVSBUJPOͷDPNNBOEʹOHJOYSFMPBEͱTMBDL௨஌Λૠೖ
    w༨ஊ
    wDPOTVMDMVTUFSʹϧʔςΟϯάؒҧ͑ͨOPEFΛೖΕͯɺΫϥελશମͷௐ
    ࢠ͕ѱ͘ͳͬͨࡍʹNJOʹճOHJOYΛSFMPBE͢Δ঱ঢ়͕ൃੜ
    wOHY@NSVCZʹSFMPBE͢ΔͱϝϞϦϦʔΫ͢Δόά͕͋ΓMPBECBMBODFS
    ่͕յʜ IJCPNB͕WBMHSJOEͰൃੜՕॴΛಛఆͯ͠ใࠂͨ͠
    
    

    View Slide

25. ಉ྅͕௚ͯ͘͠Εͨ
    IUUQTHJUIVCDPNNBUTVNPUPSOHY@NSVCZQVMM
    

    View Slide

26. ·ͱΊ
    wDPOTVMΛ࢖͏͜ͱͰWNΛଋͶͯҰͭͷγεςϜͱͯ͠ѻ͏͜ͱ͕Մೳͱͳ
    Δ
    wDPOTVMNBJOUΛ׆༻Ͱ͖ͯͳ͍ͷͰ͍͍ײ͡ʹWNΛୀආ͢ΔΈ͍ͨͷΛ࡞
    Γ͍ͨ
    wશࣗಈͳΊΒ͔αʔόʔϦιʔε੍ޚ͘ΜɺશࣗಈͳΊΒ͔δϣϒεέδϡʔ
    ϥΛ࡞͍ͬͯΔͱ͜Ζ
    w͜͏͍͏ͷΛ΍Γ͍ͨਓ͸ࠓ͙͢๻ͱϖύϥϯνϣϯˠIUUQT
    QFQBCPDPNSFDSVJUQFQBMVODIFPO
    

    View Slide