1年ほどDC/OSを運用してみた話

Transcript

1. ̍೥΄Ͳ%$04Λӡ༻ ͯ͠Έͨ࿩
2. ࣗݾ঺հ ࿨ా ږ @wappy100 גࣜձࣾαΠόʔΤʔδΣϯτ ɹΞυςΫຊ෦ ΞυςΫελδΦͷϩά෼ੳج൫ͷΠ ϯϑϥݟͯ·͢ DC/OS, k8s, Hadoop, Kafka
3. DC/OSͬͯԿʁ
4. • ίϯςφΦʔέετϨʔγϣϯΤϯδϯ • Mesosphereࣾ੡OSS • Data Center Operating System • Mesos / Marathon / Metronome / Spartan / Minuteman / …
5. ΞʔΩςΫνϟ
6. ΞʔΩςΫνϟ
7. ΞʔΩςΫνϟ
8. ΞʔΩςΫνϟ .FTPT
   NBTUFS
   ͱͳΔ
   /PEF
   Ϧιʔε੍ޚ΋Ζ΋Ζ΍ͬͯ͘ΕΔ
9. ΞʔΩςΫνϟ
10. ΞʔΩςΫνϟ ֎෦ެ։༻
    /PEF
    αʔϏεΛ֎෦ެ։͢Δࡍ͸͜ͷ
    /PEF
    Λܦ༝ͯ͠ΞΫηεͤ͞Δ
11. ΞʔΩςΫνϟ
12. ΞʔΩςΫνϟ 8PSLFS
    ͱͳΔ
    /PEF
    ΞϓϦέʔγϣϯͳͲ഑ஔ͞ΕΔͷ͸ίί
13. Πϯετʔϧ
14. DC/OSΠϯετʔϧ
15. #PPUTUSBQ
    
    /PEF DC/OSΠϯετʔϧ Πϯετʔϧॲཧ༻
    /PEF
    Λ༻ҙ
16. #PPUTUSBQ
    
    /PEF DC/OSΠϯετʔϧ 
    CPPUTUSBQ@VSM
    pMFPQUEDPT@JOTUBMM@UNQ
    DMVTUFS@OBNF
    DMVTUFSOBNF
    
    FYIJCJUPS@TUPSBHF@CBDLFOE
    TUBUJD
    JQ@EFUFDU@QVCMJD@pMFOBNF
    QBUIUPJQTDSJQU
    NBTUFS@EJTDPWFSZ
    TUBUJD
    NBTUFS@MJTU
    
    NBTUFSQSJWBUFJQ
    
    NBTUFSQSJWBUFJQ
    
    NBTUFSQSJWBUFJQ
    BHFOU@MJTU
    
    BHFOUQSJWBUFJQ
    
    BHFOUQSJWBUFJQ
    
    BHFOUQSJWBUFJQ
    
    BHFOUQSJWBUFJQ
    
    BHFOUQSJWBUFJQ
    QVCMJD@BHFOU@MJTU
    
    QVCMJDBHFOUQSJWBUFJQ
    SFTPMWFST
    
    
    
    
    TTI@QPSU
    
    TTI@VTFS
    VTFSOBNF ΫϥελఆٛϑΝΠϧΛ༻ҙ
17. #PPUTUSBQ
    
    /PEF DC/OSΠϯετʔϧ 
    DVSM
    0
    IUUQTEPXOMPBETEDPTJPEDPTTUBCMFEDPT@HFOFSBUF@DPOpHTI ಾγΣϧΛμ΢ϯϩʔυ
18. #PPUTUSBQ
    
    /PEF DC/OSΠϯετʔϧ 
    DVSM
    0
    IUUQTEPXOMPBETEDPTJPEDPTTUBCMFEDPT@HFOFSBUF@DPOpHTI
    
    TVEP
    CBTI
    EDPT@HFOFSBUF@DPOpHTI
    ŠHFODPOG ಾίϚϯυΛ࣮ߦ
19. #PPUTUSBQ
    
    /PEF DC/OSΠϯετʔϧ 
    DVSM
    0
    IUUQTEPXOMPBETEDPTJPEDPTTUBCMFEDPT@HFOFSBUF@DPOpHTI
    
    TVEP
    CBTI
    EDPT@HFOFSBUF@DPOpHTI
    ŠHFODPOG
    
    TVEP
    CBTI
    EDPT@HFOFSBUF@DPOpHTI
    ŠJOTUBMMQSFSFRT ಾίϚϯυΛ࣮ߦ
20. #PPUTUSBQ
    
    /PEF DC/OSΠϯετʔϧ 
    DVSM
    0
    IUUQTEPXOMPBETEDPTJPEDPTTUBCMFEDPT@HFOFSBUF@DPOpHTI
    
    TVEP
    CBTI
    EDPT@HFOFSBUF@DPOpHTI
    ŠHFODPOG
    
    TVEP
    CBTI
    EDPT@HFOFSBUF@DPOpHTI
    ŠJOTUBMMQSFSFRT
    
    TVEP
    CBTI
    EDPT@HFOFSBUF@DPOpHTI
    QSFqJHIU
    ಾίϚϯυΛ࣮ߦ
21. #PPUTUSBQ
    
    /PEF DC/OSΠϯετʔϧ 
    DVSM
    0
    IUUQTEPXOMPBETEDPTJPEDPTTUBCMFEDPT@HFOFSBUF@DPOpHTI
    
    TVEP
    CBTI
    EDPT@HFOFSBUF@DPOpHTI
    ŠHFODPOG
    
    TVEP
    CBTI
    EDPT@HFOFSBUF@DPOpHTI
    ŠJOTUBMMQSFSFRT
    
    TVEP
    CBTI
    EDPT@HFOFSBUF@DPOpHTI
    QSFqJHIU
    
    TVEP
    CBTI
    EDPT@HFOFSBUF@DPOpHTI
    EFQMPZ
    ಾίϚϯυΛ࣮ߦ
22. #PPUTUSBQ
    
    /PEF DC/OSΠϯετʔϧ 
    DVSM
    0
    IUUQTEPXOMPBETEDPTJPEDPTTUBCMFEDPT@HFOFSBUF@DPOpHTI
    
    TVEP
    CBTI
    EDPT@HFOFSBUF@DPOpHTI
    HFODPOG
    
    TVEP
    CBTI
    EDPT@HFOFSBUF@DPOpHTI
    JOTUBMMQSFSFRT
    
    TVEP
    CBTI
    EDPT@HFOFSBUF@DPOpHTI
    QSFqJHIU
    
    TVEP
    CBTI
    EDPT@HFOFSBUF@DPOpHTI
    EFQMPZ
    
    TVEP
    CBTI
    EDPT@HFOFSBUF@DPOpHTI
    QPTUqJHIU
    ಾίϚϯυΛ࣮ߦ
23. #PPUTUSBQ
    
    /PEF DC/OSΠϯετʔϧ IUUQ\."45&3@/0%&@*1^
    
    ʹΞΫηε
25. • ৗ࣌ىಈαʔϏε • Marathonఆٛ(json)Λॻ͍ͯσϓϩΠ Services
26. Marathonఆٛ { "id": "/test/app", "container": { "type": "DOCKER", "docker": { "image": “nginx", "network": "BRIDGE", "portMappings": [ { "containerPort": 80, "protocol": "tcp", "labels": { "VIP_0": “/test/apps:8889" } } ] } }, "instances": 3, "cpus": 1, "mem": 1024 }
27. • όονॲཧ༻εέδϡʔϥ • Metronomeఆٛ(json)Λॻ͍ͯσϓϩΠ • cronఆٛΛهड़ Jobs
28. Metronomeఆٛ { "id": "my-scheduled-job", "description": "A job that sleeps on a schedule", "run": { "cmd": "sleep 20000", "cpus": 0.01, "mem": 32, "disk": 0 }, "schedules": [ { "id": “sleep-nightly", "enabled": true, "cron": "20 0 * * *", "concurrencyPolicy": "ALLOW" } ] }
29. • ύοέʔδςϯϓϨʔτ؅ཧ • Πϯετʔϧ͢ΔͱMarathonఆٛʹै͍֤ΞϓϦέʔ γϣϯ͕σϓϩΠ͞ΕΔ • ࡉ͔͍ઃఆɺϋϯυϦϯάͳͲ೉͍͠ͷͰར༻͍ͯ͠ͳ͍ Universe
30. ӡ༻ࣄྫ ʢDC/OS 1.8.0ʙ1.9.0ʣ
31. 1. εέʔϧΞ΢τ / Πϯͷ࿩ 2. ϩά·ΘΓͷ࿩ 3. MarathonϔϧενΣοΫͷ࿩ 4. Deployment͕ऴΘΒͳ͍࿩ 5. αʔϏεσΟεΧόϦো֐ͷ࿩
32. 1. εέʔϧΞ΢τ / Πϯͷ࿩
33. ಉ྅ ʻʮෛՙࢼݧ͍ͨ͠ΜͰ ɹɹɹɹDC/OSΫϥελͰ͔͍ͨͬ͘͢͠ʯ ʮ͸͍ʯʼ ࿨ా
34. εέʔϧΞ΢τ؆୯ • bootstrap αʔόʹ͋Δ shell Λ͖࣋ͬͯͯ ৽نϗετͰ࣮ߦ͢Δ͚ͩ $ curl -O http://dcos-boot.example.com/dcos_install.sh $ sudo bash dcos_install.sh slave
35. ಉ྅ ʻʮෛՙࢼݧऴΘͬͨΜͰ ɹɹɹɹDC/OSΫϥελখ͍ͨͬ͘͢͞͠ʯ ʮ͸͍ʯʼ ࿨ా
36. ʮͲ͏΍ΔΜͩΖ͏ʯʼ ࿨ా
37. ͦΜͳ΋ͷ͸ͳ͍
38. ৽نʹ࡞Γ௚ͯ͠ blue / green ʹͨ͠
39. 2. ϩά·ΘΓͷ࿩
40. GUI͔ΒݟͯΔ
41. Metronome Job ࡞੒࣌
42. ࿨ాʻʮJobsͷ΄͏ͷϩάݟΔ͔ʯ
43. ࿨ాʻʮͲ͔͜ΒݟΔΜͩʯ
44. Marathonͷ΄͏͸ίί ίί
45. ݟΕͳ͍
46. ݟΕͳ͍
47. ݟΕͳ͍
48. ݟΕͳ͍
49. http://${MASTER_NODE}/mesos
50. Mesos Framework ͕ݟΕΔ
51. Task ͕ݟΕΔ
52. ͋ͬͨ
53. 3. MarathonϔϧενΣοΫͷ࿩
54. MarathonϔϧενΣοΫͬͯʁ { "id": “/tools/app1", "container": { "type": "DOCKER", "docker": { "image": “nginx” } }, "healthChecks": [ { "protocol": "HTTP", "path": "/health", "gracePeriodSeconds": 60, "intervalSeconds": 15, "timeoutSeconds": 10 } ], "backoffSeconds": 2, "backoffFactor": 1.2, "maxLaunchDelaySeconds": 1200, "taskKillGracePeriodSeconds": 60, • MarathonλεΫ͕ਖ਼ৗʹՔಇ͍ͯ͠Δ͔ͷϔϧε νΣοΫػߏ • httpΤϯυϙΠϯτɺίϚϯυ࣮ߦͳͲͷఆ͕ٛՄೳ
55. ಛఆͷ Marathon task ͕ Unhealty ʹͳͬͯ͠·͏ • kafka offset lag औಘ༻ʹ linkedin/burrow Λ ɹɹɹ DC/OS ্Ͱϗετ͍ͯͨ͠ • ͍͔ͭΒ͔ Unhealty ͱͳ͍ͬͯͨ ίί
56. ಛఆͷ Marathon task ͕ Unhealty ʹͳͬͯ͠·͏ • Marathon ϔϧενΣοΫ༻ΤϯυϙΠϯτΛɺϝτϦΫε औಘ༻ͷΤϯυϙΠϯτͱͯ͠͠·͍ͬͯͨ http://{burrow}/metrics • kafka topic ͕૿͑ΔͨͼʹϨεϙϯεαΠζ͕૿͑ɺ Marathon ϔϧενΣοΫ͕ΤϥʔͱͳΔ https://github.com/mesosphere/marathon/issues/3289 • ϔϧενΣοΫ༻ͷΤϯυϙΠϯτΛ༻ҙͯ͠ରԠ
57. 4. Deployment͕ऴΘΒͳ͍࿩
58. Deploymentͬͯʁ • DC/OS΁౤͛ͨMarathonఆٛΛαϒϛοτ͢Δॲཧ • MarathonλεΫ͕૸ͬͨΒ׬ྃ • CPU / Memory Ϧιʔε͕଍Γͳ͍৔߹ʹ Pending ʹ ͳͬͨΓ͢Δ
59. ऴΘΒͳ͍ • image pull ࣦഊΈ͍ͨͳ task Failed ͕ى͖͍ͯΔΘ͚Ͱ ΋ͳ͍ • Ϧιʔε΋଍Γ͍ͯΔ • dcos marathon deployment stop (CLIπʔϧ) ΋ޮ͔ͳ͍
60. ௚઀MarathonΛݟΔ • http://${MASTER_NODE_IP}/marathon/ui/ • task / deployment ͷ restart / destroy ΋ޮՌͳ͠ • destroy ͯ͠΋ suspended task Ͱ࢒ͬͯ͠·͏ɾɾ
61. ίϯςφ͕৐͍ͬͯΔϗετΛݟΔ • docker ίϯςφ͕ϋϯάͯ͠ docker stop / kill / rm ͕ޮ͔ͳ͍ঢ়ଶ • deployment ͸ίϯςφͷ࡟আ଴ͪͰਐΜͰ͍ͳ͍༷ࢠ • docker rm -f Ͱڧ੍࡟আ͢Δ΋ mesos ͷ task ͱͯ͠͸࢒͍ͬͯΔ • ࠶౓Deployͯ͠΋ docker stop ॲཧ͔Β࢝·ͬͯ͠·͏ • Marathon task ID Λผͷ΋ͷʹͯ͠Deploy͢Δ͜ͱͰରॲͨ͠
62. 5. αʔϏεσΟεΧόϦো֐ͷ࿩
63. MarathonλεΫఆٛͰϩʔυόϥϯ εΛ࣮૷͢ΔͱɺDC/OSͰαʔϏε σΟεΧόϦͯ͘͠ΕΔ
64. Marathonఆٛ { "id": "/test/apps", "container": { "type": "DOCKER", "docker": { "image": “nginx", "network": "BRIDGE", "portMappings": [ { "containerPort": 80, "protocol": "tcp", "labels": { "VIP_0": “/test/apps:8889" } } ] } }, "instances": 3, "cpus": 1, "mem": 1024 }
65. Marathonఆٛ { "id": "/test/apps", "container": { "type": "DOCKER", "docker": { "image": “nginx", "network": "BRIDGE", "portMappings": [ { "containerPort": 80, "protocol": "tcp", "labels": { "VIP_0": “/test/apps:8889" } } ] } }, "instances": 3, "cpus": 1, "mem": 1024 }
66. Lookup # dig testapps.marathon.l4lb.thisdcos.directory TFSWJDFOBNFNBSBUIPOMMCUIJTEDPTEJSFDUPSZ
    Ͱ%$04಺෦͔ΒMPPLVQՄೳͱͳΔ
67. Lookup # dig testapps.marathon.l4lb.thisdcos.directory ; <<>> DiG 9.9.4-RedHat-9.9.4-29.el7_2.2 <<>> testapps.marathon.l4lb.thisdcos.directory ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13203 ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;testapps.marathon.l4lb.thisdcos.directory. IN A ;; ANSWER SECTION: testapps.marathon.l4lb.thisdcos.directory. 5 IN A 11.219.114.140 ;; Query time: 1 msec ;; SERVER: 198.51.100.1#53(198.51.100.1) ;; WHEN: Sun Nov 05 05:54:38 UTC 2017 ;; MSG SIZE rcvd: 86
68. Lookup # curl testapps.marathon.l4lb.thisdcos.directory:8889 <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1>
69. ͍͚ͦ͏ʂ ͱࢥͬͯ͠͹Β͘ӡ༻ͯͨ͠Βো֐ൃੜ
70. Το # curl prd-ui.marathon.l4lb.thisdcos.directory:3000 curl: (6) Could not resolve host: prd-ui.marathon.l4lb.thisdcos.directory
71. Τοοο # dig prd-ui.marathon.l4lb.thisdcos.directory ;; ANSWER SECTION: prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58 prd-ui.marathon.l4lb.thisdcos.directory. 5 IN A 11.214.212.58
72. ͠͹Β͘͢ΔͱANSWER͢Βฦ͞ͳ͘ͳΔ # dig prd-ui.marathon.l4lb.thisdcos.directory ; <<>> DiG 9.9.4-RedHat-9.9.4-38.el7_3.2 <<>> prd-ui.marathon.l4lb.thisdcos.directory ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 55428 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;prd-ui.marathon.l4lb.thisdcos.directory. IN A ;; AUTHORITY SECTION: thisdcos.directory. 130 IN SOA ns-1461.awsdns-54.org. awsdns- hostmaster.amazon.com. 1 7200 900 1209600 86400 ;; Query time: 1 msec ;; SERVER: 198.51.100.1#53(198.51.100.1) ;; WHEN: Sun Nov 05 15:31:03 JST 2017 ;; MSG SIZE rcvd: 153
73. ͠͹Β͘͢ΔͱANSWER͢Βฦ͞ͳ͘ͳΔ # dig prd-ui.marathon.l4lb.thisdcos.directory ; <<>> DiG 9.9.4-RedHat-9.9.4-38.el7_3.2 <<>> prd-ui.marathon.l4lb.thisdcos.directory ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 55428 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;prd-ui.marathon.l4lb.thisdcos.directory. IN A ;; AUTHORITY SECTION: thisdcos.directory. 130 IN SOA ns-1461.awsdns-54.org. awsdns- hostmaster.amazon.com. 1 7200 900 1209600 86400 ;; Query time: 1 msec ;; SERVER: 198.51.100.1#53(198.51.100.1) ;; WHEN: Sun Nov 05 15:31:03 JST 2017 ;; MSG SIZE rcvd: 153
74. ͠͹Β͘͢ΔͱANSWER͢Βฦ͞ͳ͘ͳΔ # dig prd-ui.marathon.l4lb.thisdcos.directory ; <<>> DiG 9.9.4-RedHat-9.9.4-38.el7_3.2 <<>> prd-ui.marathon.l4lb.thisdcos.directory ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 55428 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;prd-ui.marathon.l4lb.thisdcos.directory. IN A ;; AUTHORITY SECTION: thisdcos.directory. 130 IN SOA ns-1461.awsdns-54.org. awsdns- hostmaster.amazon.com. 1 7200 900 1209600 86400 ;; Query time: 1 msec ;; SERVER: 198.51.100.1#53(198.51.100.1) ;; WHEN: Sun Nov 05 15:31:03 JST 2017 ;; MSG SIZE rcvd: 153 ίϨ͸4QBSUBO %/4
    1SPYZ
75. Ͳ͏͔ͨ͠ • Load Balancing / VIP ൃߦ͍ͯ͠ΔMinutemanαʔϏε Λ࠶ىಈ͢Δ͜ͱͰରԠ • DC/OS 1.9.0 ʹόʔδϣϯόοϓͨ͠Β࠶ൃ͠ͳ͘ͳͬͨ ʢݪҼ௥͑ͣʣ # systemctl restart dcos-minuteman.service # dig prd-ui.marathon.l4lb.thisdcos.directory +short 11.214.212.58
76. ·ͱΊ • Mesos / Marathon / Metronome ·ΘΓͷ ஌͕ࣝඞཁ • ಺෦ίϯϙʔωϯτͷτϥϒϧγϡʔτ͕ ͭΒ͍ • k8sͱൺ΂Δͱ΍͸Γ৘ใྔ͕ɾɾɾ
77. ·ͱΊ • ίϯςφΦʔέετϨʔγϣϯΤϯδϯ ͱͯ͠͸ػೳ͸े෼ɺߏங΋؆୯ • ̍೥Ҏ্ӡ༻͍ͯ͠Δ͕ΫϥελམͪΔ ͳͲͷக໋తͳো֐ʹ͸ૺ۰ͤͣʢͨͩ ͠ϗετμ΢ϯʹ΋ૺ۰͍ͯ͠ͳ͍ʣ
78. ͓ΘΓ