Share
Embed
1年ほどDC/OSを運用してみた話
wappy100
November 15, 2017
Transcript
78 slides
-
• ίϯςφΦʔέετϨʔγϣϯΤϯδϯ • MesosphereࣾOSS • Data Center Operating System •
Mesos / Marathon / Metronome / Spartan / Minuteman / … -
#PPUTUSBQ /PEF DC/OSΠϯετʔϧ CPPUTUSBQ@VSMpMFPQUEDPT@JOTUBMM@UNQ DMVTUFS@OBNFDMVTUFSOBNF FYIJCJUPS@TUPSBHF@CBDLFOETUBUJD JQ@EFUFDU@QVCMJD@pMFOBNFQBUIUPJQTDSJQU NBTUFS@EJTDPWFSZTUBUJD NBTUFS@MJTU
NBTUFSQSJWBUFJQ NBTUFSQSJWBUFJQ NBTUFSQSJWBUFJQ BHFOU@MJTU BHFOUQSJWBUFJQ BHFOUQSJWBUFJQ BHFOUQSJWBUFJQ BHFOUQSJWBUFJQ BHFOUQSJWBUFJQ QVCMJD@BHFOU@MJTU QVCMJDBHFOUQSJWBUFJQ SFTPMWFST TTI@QPSU TTI@VTFSVTFSOBNF ΫϥελఆٛϑΝΠϧΛ༻ҙ -
Marathonఆٛ { "id": "/test/app", "container": { "type": "DOCKER", "docker": {
"image": “nginx", "network": "BRIDGE", "portMappings": [ { "containerPort": 80, "protocol": "tcp", "labels": { "VIP_0": “/test/apps:8889" } } ] } }, "instances": 3, "cpus": 1, "mem": 1024 } -
Metronomeఆٛ { "id": "my-scheduled-job", "description": "A job that sleeps on
a schedule", "run": { "cmd": "sleep 20000", "cpus": 0.01, "mem": 32, "disk": 0 }, "schedules": [ { "id": “sleep-nightly", "enabled": true, "cron": "20 0 * * *", "concurrencyPolicy": "ALLOW" } ] } -
1. εέʔϧΞτ / Πϯͷ 2. ϩά·ΘΓͷ 3. MarathonϔϧενΣοΫͷ 4. Deployment͕ऴΘΒͳ͍
5. αʔϏεσΟεΧόϦোͷ -
εέʔϧΞτ؆୯ • bootstrap αʔόʹ͋Δ shell Λ͖࣋ͬͯͯ ৽نϗετͰ࣮ߦ͢Δ͚ͩ $ curl -O
http://dcos-boot.example.com/dcos_install.sh $ sudo bash dcos_install.sh slave -
MarathonϔϧενΣοΫͬͯʁ { "id": “/tools/app1", "container": { "type": "DOCKER", "docker": {
"image": “nginx” } }, "healthChecks": [ { "protocol": "HTTP", "path": "/health", "gracePeriodSeconds": 60, "intervalSeconds": 15, "timeoutSeconds": 10 } ], "backoffSeconds": 2, "backoffFactor": 1.2, "maxLaunchDelaySeconds": 1200, "taskKillGracePeriodSeconds": 60, • MarathonλεΫ͕ਖ਼ৗʹՔಇ͍ͯ͠Δ͔ͷϔϧε νΣοΫػߏ • httpΤϯυϙΠϯτɺίϚϯυ࣮ߦͳͲͷఆ͕ٛՄೳ -
ಛఆͷ Marathon task ͕ Unhealty ʹͳͬͯ͠·͏ • kafka offset lag
औಘ༻ʹ linkedin/burrow Λ ɹɹɹ DC/OS ্Ͱϗετ͍ͯͨ͠ • ͍͔ͭΒ͔ Unhealty ͱͳ͍ͬͯͨ ίί -
ಛఆͷ Marathon task ͕ Unhealty ʹͳͬͯ͠·͏ • Marathon ϔϧενΣοΫ༻ΤϯυϙΠϯτΛɺϝτϦΫε औಘ༻ͷΤϯυϙΠϯτͱͯ͠͠·͍ͬͯͨ
http://{burrow}/metrics • kafka topic ͕૿͑ΔͨͼʹϨεϙϯεαΠζ͕૿͑ɺ Marathon ϔϧενΣοΫ͕ΤϥʔͱͳΔ https://github.com/mesosphere/marathon/issues/3289 • ϔϧενΣοΫ༻ͷΤϯυϙΠϯτΛ༻ҙͯ͠ରԠ -
ऴΘΒͳ͍ • image pull ࣦഊΈ͍ͨͳ task Failed ͕ى͖͍ͯΔΘ͚Ͱ ͳ͍ •
ϦιʔεΓ͍ͯΔ • dcos marathon deployment stop (CLIπʔϧ) ޮ͔ͳ͍ -
MarathonΛݟΔ • http://${MASTER_NODE_IP}/marathon/ui/ • task / deployment ͷ restart /
destroy ޮՌͳ͠ • destroy ͯ͠ suspended task Ͱͬͯ͠·͏ɾɾ -
ίϯςφ͕͍ͬͯΔϗετΛݟΔ • docker ίϯςφ͕ϋϯάͯ͠ docker stop / kill / rm
͕ޮ͔ͳ͍ঢ়ଶ • deployment ίϯςφͷআͪͰਐΜͰ͍ͳ͍༷ࢠ • docker rm -f Ͱڧ੍আ͢Δ mesos ͷ task ͱ͍ͯͬͯ͠Δ • ࠶Deployͯ͠ docker stop ॲཧ͔Β࢝·ͬͯ͠·͏ • Marathon task ID Λผͷͷʹͯ͠Deploy͢Δ͜ͱͰରॲͨ͠ -
Marathonఆٛ { "id": "/test/apps", "container": { "type": "DOCKER", "docker": {
"image": “nginx", "network": "BRIDGE", "portMappings": [ { "containerPort": 80, "protocol": "tcp", "labels": { "VIP_0": “/test/apps:8889" } } ] } }, "instances": 3, "cpus": 1, "mem": 1024 } -
Marathonఆٛ { "id": "/test/apps", "container": { "type": "DOCKER", "docker": {
"image": “nginx", "network": "BRIDGE", "portMappings": [ { "containerPort": 80, "protocol": "tcp", "labels": { "VIP_0": “/test/apps:8889" } } ] } }, "instances": 3, "cpus": 1, "mem": 1024 } -
Lookup # dig testapps.marathon.l4lb.thisdcos.directory ; <<>> DiG 9.9.4-RedHat-9.9.4-29.el7_2.2 <<>> testapps.marathon.l4lb.thisdcos.directory
;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13203 ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;testapps.marathon.l4lb.thisdcos.directory. IN A ;; ANSWER SECTION: testapps.marathon.l4lb.thisdcos.directory. 5 IN A 11.219.114.140 ;; Query time: 1 msec ;; SERVER: 198.51.100.1#53(198.51.100.1) ;; WHEN: Sun Nov 05 05:54:38 UTC 2017 ;; MSG SIZE rcvd: 86 -
Lookup # curl testapps.marathon.l4lb.thisdcos.directory:8889 <!DOCTYPE html> <html> <head> <title>Welcome to
nginx!</title> <style> body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> -
Το # curl prd-ui.marathon.l4lb.thisdcos.directory:3000 curl: (6) Could not resolve host:
prd-ui.marathon.l4lb.thisdcos.directory -
Τοοο # dig prd-ui.marathon.l4lb.thisdcos.directory ;; ANSWER SECTION: prd-ui.marathon.l4lb.thisdcos.directory. 5 IN
A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 -
͠Β͘͢ΔͱANSWER͢Βฦ͞ͳ͘ͳΔ # dig prd-ui.marathon.l4lb.thisdcos.directory ; <<>> DiG 9.9.4-RedHat-9.9.4-38.el7_3.2 <<>> prd-ui.marathon.l4lb.thisdcos.directory
;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 55428 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;prd-ui.marathon.l4lb.thisdcos.directory. IN A ;; AUTHORITY SECTION: thisdcos.directory. 130 IN SOA ns-1461.awsdns-54.org. awsdns- hostmaster.amazon.com. 1 7200 900 1209600 86400 ;; Query time: 1 msec ;; SERVER: 198.51.100.1#53(198.51.100.1) ;; WHEN: Sun Nov 05 15:31:03 JST 2017 ;; MSG SIZE rcvd: 153 -
͠Β͘͢ΔͱANSWER͢Βฦ͞ͳ͘ͳΔ # dig prd-ui.marathon.l4lb.thisdcos.directory ; <<>> DiG 9.9.4-RedHat-9.9.4-38.el7_3.2 <<>> prd-ui.marathon.l4lb.thisdcos.directory
;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 55428 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;prd-ui.marathon.l4lb.thisdcos.directory. IN A ;; AUTHORITY SECTION: thisdcos.directory. 130 IN SOA ns-1461.awsdns-54.org. awsdns- hostmaster.amazon.com. 1 7200 900 1209600 86400 ;; Query time: 1 msec ;; SERVER: 198.51.100.1#53(198.51.100.1) ;; WHEN: Sun Nov 05 15:31:03 JST 2017 ;; MSG SIZE rcvd: 153 -
͠Β͘͢ΔͱANSWER͢Βฦ͞ͳ͘ͳΔ # dig prd-ui.marathon.l4lb.thisdcos.directory ; <<>> DiG 9.9.4-RedHat-9.9.4-38.el7_3.2 <<>> prd-ui.marathon.l4lb.thisdcos.directory
;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 55428 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;prd-ui.marathon.l4lb.thisdcos.directory. IN A ;; AUTHORITY SECTION: thisdcos.directory. 130 IN SOA ns-1461.awsdns-54.org. awsdns- hostmaster.amazon.com. 1 7200 900 1209600 86400 ;; Query time: 1 msec ;; SERVER: 198.51.100.1#53(198.51.100.1) ;; WHEN: Sun Nov 05 15:31:03 JST 2017 ;; MSG SIZE rcvd: 153 ίϨ4QBSUBO %/41SPYZ -
Ͳ͏͔ͨ͠ • Load Balancing / VIP ൃߦ͍ͯ͠ΔMinutemanαʔϏε Λ࠶ىಈ͢Δ͜ͱͰରԠ • DC/OS
1.9.0 ʹόʔδϣϯόοϓͨ͠Β࠶ൃ͠ͳ͘ͳͬͨ ʢݪҼ͑ͣʣ # systemctl restart dcos-minuteman.service # dig prd-ui.marathon.l4lb.thisdcos.directory +short 11.214.212.58 -
·ͱΊ • Mesos / Marathon / Metronome ·ΘΓͷ ͕ࣝඞཁ •
෦ίϯϙʔωϯτͷτϥϒϧγϡʔτ͕ ͭΒ͍ • k8sͱൺΔͱΓใྔ͕ɾɾɾ