Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Bloxが切り開くECSの世界

 Bloxが切り開くECSの世界

第二回 合同勉強会 in 福岡 の発表資料です。

https://connpass.com/event/60964/

Koji Nakayama

August 26, 2017
Tweet

More Decks by Koji Nakayama

Other Decks in Technology

Transcript

  1. ᛔ૩奧Օ • Ӿઊ ଛလ • μ϶φϮϊϐϖ ϯϝαϸίϤϷςЄϠφ᮱ AWSϊϷϲЄτϴЀίЄκϓμϕ • AWSΨڥአͭ͵αЀϢ϶΄戔懯/䯤塈

    • GitHub: knakayama • 奺䵉 • ηЀϤϹςЄϝ΄晁አ3ଙ • AWSΚΠ͵͡͹͵΄ͽμ϶φϮϊϐϖ΁敢实 • ๋ᬪϯϝαϸίϤϷςЄϠφ᮱Ύ吖㵕 2
  2. ςЄϝϹφ is • ϜϫΡ䦒΁΅๋ṛ ! • ͽΘݻͣӧݻͣ͘ΠΔͯΞ΃ • ϢϹЄϭϼЄμֵ͚͵͚ΩͽͯͧͿ •

    RDSֵ͚͵͚ΩͽͯͧͿ • Ϳͭ͜ͼΘګᴴ͢… • ϫϚЄυϖςЄϠφ΀΄ͽϳЄσ㯎ͽͽͣΡͩ;͢ᴴΟ΢Ρ • ق᮱Lambdaضኞ΁͠氎͚ͽͣ͵Ο๋ṛͶͧͿ柞΄䔺ԃ΅΀͚ 8
  3. ECS΄͜΢͚ͭ;ͩΣ • ϳЄσͽᓕቘͽͣΡ᮱ړ;AWS΁͠Δ͡ͱͽͣΡ᮱ړ΄䂜໶͢Ξ͚ • Devፓ娄 • πЀϓϗͽ㵕֢ͫͱΡϊЄφπЄϖ;DockerfileΨᓕቘͯ΢Ά͚͚ • Opsፓ娄 •

    ϔϤϺα͢䮭 → αϮЄυ֢͹ͼϔϤϺαͯ΢ΆECS͢Ξͭ΀΁Κ͹ͼͥ΢Ρ • CloudFormationͽᓕቘͽͣΡ;ͩΣΘΞ͚ • ٖ᮱ጱ΁EC2ֵ͹ͼΡ΄ͽϳЄσ㯎ͽϷϊЄφ䃀ΚͱΡ • Lambda/API GatewayͶ͹͵ΟͿ͜΀ΡҘ • ECSᛔ֛΅僻ා 9
  4. ECS΄ͫ͠Ο͚(ͳ΄1) • πЀϓϗαЀφόЀφ • Amazon ECS container agent͢㵕֢ͯΡEC2αЀφόЀφ • πЀϓϗαЀφόЀφӤͽDockerπЀϓϗ͢㵕֢ͯΡ

    • Ӟᛱጱ΁΅Amazon ECS container agent͢αЀφϕЄϸͫ΢͵ECS-optimized AMI͡Ο᩸㵕ͫͱΡ • च๜ጱ΁DockerπЀϓϗ͢ڥአͯΡϷϊЄφ΄ϤЄϸ;ͭͼᘍ͞ͼᜉ͚ • μ϶φό • 1ͺզӤ΄πЀϓϗαЀφόЀφᮭ • DockerπЀϓϗ΅μ϶φόٖ΄ϷϊЄφ(CPU/ϮϯϷ΀Ϳ)Ψڥአͭͼ䋚ᤈͫ΢Ρ 11
  5. ECS΄Ԇᥝ΀༷ஷ(ͳ΄4) • ςЄϠφ • ECS΁ͧ͠ΡφξυϲЄ϶(;ᘍ͞ͼΘय़ͣͥϜχϹͼ΀͚) • ςЄϠφΨ᭗ͭͼόφμΨ᩸㵕ͫͱΡͩ;ͽόφμΨ๗இͭ͵ᇫ䙪΁φξυϲЄϷЀνͭͼͥ΢Ρ • ͚ͥͺόφμΨ᩸㵕ͫͱΡ͡/Ϳ΄πЀϓϗαЀφόЀφ΁όφμΨᯈᗝͯΡ͡΀Ϳ •

    όφμᯈᗝ䜐ኼ/όφμᯈᗝګᴴ • ςЄϠφ͡Ο᩸㵕ͯΡόφμΨͿ͚͜͹͵ϸЄϸͽμ϶φόӤ΁ᯈᗝͯΡ͡ • μ϶φό΄ϷϊЄφၾ揲ᰁ๋͢ੜ΁΀ΡΞ͜΁ͭ͵͚(όφμᯈᗝ䜐ኼ) • ṛ͚αЀφόЀφόαϤ΄πЀϓϗαЀφόЀφ΁΄Εᯈᗝͫͱ͵͚(όφμᯈᗝګᴴ) • ՜΁Θ奞ͥ͡೰ਧݢᚆ 15
  6. ECS΄Ԇᥝ΀༷ஷ(ͳ΄5) • όЄοϐϕνϸЄϤ(㷗ੂ΁΅ALB΄༷ஷ) • ALB͡Οόφμ΁ϕ϶ϢΰϐμΨϸЄϓΰЀνͯΡ͵Η΄㶨֖ • όЄοϐϕνϸЄϤ΅ςЄϠφ;樛昧՞ͧͫ΢ݱόφμ΁ϕ϶Ϣΰϐμ͢ၞ΢Ρ • ALB΅όЄοϐϕνϸЄϤΎ΄ϥϸφώδϐμΨ᭗ͭͼςЄϠφϔΰφθϝϷΨ䋚匍ͯΡ •

    ύαϗϬϐμϨφϕϪЄϕϫϐϡЀν • πЀϓϗαЀφόЀφ;όφμͽڥአͯΡϪЄϕΨ㵕ጱ΁ϫϐϡЀνͭͼͥ΢Ρ䱛ᚆ • ͩ΢͢΀͚;ݶͮϪЄϕΨڥአͯΡόφμ͢ݶӞπЀϓϗαЀφόЀφͽ᩸㵕ͽͣ΀͚ • CLB΅䌏䖕ͭͼ΀͚ 16
  7. ECS΄Ԇᥝ΀༷ஷ(ͳ΄6) • ECS Events • ECSٖ΄ᇫ䙪΁䄜۸᩸ͣ͢͵䁰ݳ΁CloudWatch EventsΎͫΔͬΔ΀ αϦЀϕΨᭆ͹ͼͥ΢Ρ • πЀϓϗαЀφόЀφ/όφμ᩸͢㵕

    or ؊ྊͭ͵/etc... • ՜΄AWSςЄϠφΨόЄοϐϕ΁ͯΡͩ;ͽαϦЀϕϖϷϣЀ΁㳌 ቘ͢ݢᚆ • Slack΁᭗ᎣͯΡ/ෛ͚ͭόφμΨ᩸㵕ͫͱΡ/etc... 18
  8. v0.3.0;v1.0΄晅͚ • v0.3.0΅ϊЄφπЄϖل樄ͭ͵͡ΟΕΩ΀ͽ樄咲ͭͼ͚ͩ͜Ͳ;͚͜ఽͮ • Go䋚ᤰ • Bloxአ΄厏हΨᛔړͽ֢Ρ஠ᥝ͘͢Ρ • Deprecated •

    v1.0΅ϫϚЄυϖςЄϠφΨፓ೰ͭͼ͚Ρ • ͺΔΠ ECS΄䱛ᚆ;ͭͼ൉׀ͫ΢Ρ (Ԩਧ) • ϫϚυϮЀϕπЀϊЄϸ/CLI/SDKs͡Ο඙֢ݢᚆ(Ԩਧ) • Java䋚ᤰ https://github.com/blox/blox/blob/dev/FAQ.md 21
  9. όφμᯈᗝ䜐ኼ 䜐ኼ ఺ޱ binpack μ϶φό΄CPU/ϮϯϷၾ揲ᰁ๋͢ ੜ;΀ΡΞ͜΁όφμΨᯈᗝ random μ϶φό΁όφμΨ϶Ѐύϭͽᯈᗝ spread ೰ਧͭ͵ં௔

    / instanceId / host ࣐͢ᒵ΁΀ΡΞ͜όφμΨᯈᗝ http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-placement-strategies.html 25
  10. όφμᯈᗝګᴴ ګᴴ ఺ޱ distinctInstance όφμ͢吖΀ΡπЀϓϗαЀφ όЀφ΁ᯈᗝͫ΢ΡΞ͜ګᴴ memberOf Cluster Query Languageͽᤒ匍

    ͭ͵ୗΨ伋͵ͯΞ͜΁όφμ΄ ᯈᗝΨګᴴ http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-placement-constraints.html 26
  11. ͳ΄՜ • ςЄϠφ΄Desired Countͽ૶๕ͯΡόφμහΨ姘೮ • ALB/CLB΄ϥϸφώδϐμΨڥአͭ͵ςЄϠφϔΰφθϝϷ • CloudWatch Metrics΁चͻ͚͵ςЄϠφ΄ηЄϕφξЄϷЀν •

    RunTask or StartTask APIͽձ఺΄όαϬЀνͽόφμΨ᩸㵕 • cron୵ୗͽᇙਧ΄䦒樌΁όφμΨ᩸㵕 http://docs.aws.amazon.com/AmazonECS/latest/developerguide/scheduling_tasks.html 27
  12. v0.3.0΄༷ᥝ • μ϶φό΄ᇫ䙪ᓕቘ;ςЀϤϸ;΀ΡφξυϲЄ϶͢ݶ䬕 • cluster-state service → ᇫ䙪ᓕቘ • daemon-scheduler

    → φξυϲЄ϶ • ͩ΢ΟΨREST APIͽ඙֢ݢᚆ • ϺЄθϸPC/AWS΁ϔϤϺαݢᚆ • DockerαϮЄυΘአ఺ͭͼͥ΢ͼ͚Ρ • ᛔړͽGoΨπЀϞαϸͯΡͩ;ΘݢᚆͶ͢πЀϓϗͽ㵕ͯ͡;͠ಋ敽 • 厏हΨ䯤塈ͯΡ͵Η΄ςЀϤϸCloudFormationϓЀϤϹЄϕΘ͘Ρ • DeprecatedͶ༷͢ஷᛔ֛΅v1.0;֒ͼ͚Ρ΄ͽ݇ᘍ΁΀Ρ 32
  13. cluster-state service • μ϶φό΄ᇫ䙪ΨϷίϸόαϭ΁ᓕቘ • μ϶φό΁ෛ͚ͭπЀϓϗαЀφόЀφ/όφμ᩸͢㵕 or ؊ྊͭ͵΀Ϳ • ͩ΢Ο΄ఘ䁭Ψetcd΁כਂ

    • cluster-state service΄ఘ䁭Ψڥአͭͼᛔړ΄厏ह΁ݳ͹͵φξυϲЄ϶͢䋚ᤰݢᚆ • 㶨֛ͽ΅μ϶φό΄ᇫ䙪ᓕቘͭͭ͡΀͚ • Ϸίϸόαϭ΄ఘ䁭㷧ᵞ΁ECS EventsΨڥአͭͼ͚Ρ • ECS Events΄όЄοϐϕ΁೰ਧͫ΢͵SQS͡ΟϮϐψЄυΨݐ஑ͭͼetcd΁כਂ • CloudWatch Events/SQS/etcd΅㳨᭔አ఺ͯΡ஠ᥝ͘Π https://github.com/blox/blox/tree/v0.3/cluster-state-service 33
  14. deamon-scheduler • πЀϓϗαЀφόЀφ䶅΁1ͺ΄όφμΨᯈᗝͯΡφξυϲЄ϶ • 哶憙/Ϻν㷧ᵞᔮ΄όφμ(Datadog/Fluentd΀Ϳ)ΨᓕቘͯΡ͵Η΁ֵ͞Ρ • ݱπЀϓϗαЀφόЀφ䶅΁όφμ᩸͢㵕ͭͼ͚Ρ஠ᥝ͘͢Ρ͵Η • ٖ᮱ጱ΁cluster-state serviceΨڥአͭͼ͚Ρ

    • ϷϢήϹЀφϯϔϸ;͚֖͜ᗝ՞ͧ • μ϶φό΁ෛ͚ͭπЀϓϗαЀφόЀφ͢ጭ梍ͫ΢͵ΟόφμΨ᩸㵕ͫͱΡ • ᭋ΁πЀϓϗαЀφόЀφ͢ڷᴻͫ΢͵ΟόφμΘڷᴻͯΡ https://github.com/blox/blox/tree/v0.3/daemon-scheduler 34
  15. daemon-scheduler΄Ԇᥝ΀༷ஷ • Environment • όφμਧ嬝;μ϶φόΨΈΘ՞ͧ͵ఘ䁭 • Environment΄ᇫ䙪Ψdaemon-scheduler͢姘೮ͭΞ͜;ͭͼͥ΢Ρ • Environment΅ͥ͘Δͽఘ䁭ͽ͘Πզӥ΄DeploymentΨ䋚ෞͭͼতΗͼECS΁όφμ ᩸͢㵕ͯΡ

    • Deployment • EnvironmentΨμ϶φόӤ΁ϔϤϺαͯΡ඙֢ • DeploymentΨ᭗ͭͼEnvironment͢μ϶φόӤ΁઀樄ͫ΢Ρ https://github.com/blox/blox/tree/v0.3/daemon-scheduler 35
  16. cluster-state service΄REST API Path(Base Path΅ /v1) Method 㳌ቘ /instances GET

    قͼ΄πЀϓϗαЀφόЀφΨᤒᐏ /tasks GET قͼ΄όφμΨᤒᐏ /instances/{cluster}/{arn} GET {cluster} ٖ΄ {arn} ͽ೰ਧͫ΢͵πЀ ϓϗαЀφόЀφΨᤒᐏ /tasks/{cluster}/{arn} GET {cluster} ٖ΄ {arn} ͽ೰ਧͫ΢͵όφ μΨᤒᐏ /stream/instances GET ϞφϞ϶ϮЄόͽ೰ਧͫ΢͵αЀφό Ѐφ΄ఘ䁭ΨφϕϷЄϭͽᤒᐏ /stream/tasks GET ϞφϞ϶ϮЄόͽ೰ਧͫ΢͵όφμ΄ ఘ䁭ΨφϕϷЄϭͽᤒᐏ https://github.com/blox/blox/blob/v0.3/cluster-state-service/swagger/v1/swagger.json 38
  17. deamon-scheduler΄REST API Path(Base Path΅ /v1) Method 㳌ቘ /ping GET HTTPφϓЄόφπЄϖ200Ψᬬᒼ

    /environments GET قͼ΄EnvironmentΨᤒᐏ /environments POST EnvironmentΨ֢౮ /environments/{name} GET {name} ͽ೰ਧͫ΢͵EnvironmentΨᤒᐏ /environments/{name} DELETE {name} ͽ೰ਧͫ΢͵EnvironmentΨڷᴻ /environments/{name}/deployments GET {name} ͽ೰ਧͫ΢͵Environment΁戔ਧͫ΢ ͼ͚Ρقͼ΄DeploymentΨᤒᐏ /environments/{name}/deployments POST {name} ͽ೰ਧͫ΢͵Environment΁䌏ͭͼ DeploymentΨ֢౮ /environments/{name}/deployments/{id} GET {name} ͽ೰ਧͫ΢͵Environment΁戔ਧͫ΢ ͼ͚Ρ {id} ;͚͜挷㳨ৼ΄DeploymentΨᤒᐏ https://github.com/blox/blox/blob/v0.3/daemon-scheduler/swagger/v1/swagger.json 39
  18. Environment΄ఘ䁭 $ curl -s 127.0.0.1:2000/v1/environments/test | jq { "deploymentToken": "c6f05550-0c2a-42b9-aa81-f66c3c60853e",

    "health": "healthy", "instanceGroup": { "cluster": "arn:aws:ecs:ap-northeast-1:111122223333:cluster/blox2" }, "name": "test", "taskDefinition": "arn:aws:ecs:ap-northeast-1:111122223333:task-definition/nginx:37" } 40
  19. Deployment΄ఘ䁭 $ curl -s 127.0.0.1:2000/v1/environments/test/deployments | jq { "items": [

    { "environmentName": "test", "failedInstances": [], "id": "95d3644f-3624-4324-81fe-35d50583f420", "status": "completed", "taskDefinition": "arn:aws:ecs:ap-northeast-1:111122223333:task-definition/nginx:37" } ] } 41
  20. v1.0΄༷ᥝ • AWSӤͽ㵕֢ͯΡECS΄ϫϚЄυϖθφόϭφξυϲЄ϶ • ECS΄䱛ᚆ;ͭͼ൉׀ͫ΢Ρ΄ͽϳЄσ΅BloxΨԪڹ΁䯤塈ͯΡ஠ᥝ͢΀͚ • Ӟ䖕ϳЄσ΄AWS厏हͽΘ㵕͡ͱΡ΄ͽᇿᛔ΁θφόϫαχͯΡͩ;Θݢᚆ • ΔͶ戔懯΁樛ͯΡϖκϲϮЀϕ͘͢ΡͶͧͽ䋚ᤰ΅Α;ΩͿͽͣͼ΀͚ •

    v0.3.0;ݶͮΞ͜΁ΔͰ΅daemon-schedulerΨ䋚ᤰ • Ք஍՜΄φξυϲЄ϶Ψ᭄ےͭͼ͚ͥԨਧ • ECS΄ϊЄφπЄϖ΄Ӟ᮱ΨApache 2.0ͽل樄ͭͼ͚ΡҘ • v1.0΄֖ᗝ՞ͧ͢ྋፗΞͥړ͡Ο΀͚ ! https://github.com/blox/blox/blob/dev/FAQ.md 44
  21. Blox frontend • BloxΨ඙֢ͯΡ͵Η΄HTTPεЀϖϪαЀϕ • API Gateway;Lambda͡Ο䯤౮ͫ΢Ρ • AWS Proxyͽጭ梍ͫ΢͵Lambda͢䋚檭΄㳌ቘΨ䋚ෞ

    • Invokeͫ΢͵Lambda΅Data ServiceΨ޷ΉڊͭDynamoDBΎϔЄόΨכਂ • Ϯϊϐϖ䶅΁㳨΄Lambda΁ͯΡ΄ͽ΅΀ͥق᮱1ͺ΄Lambdaͽ㳌ቘͫͱΡ • API Gateway;΅IAMͽ扯戣ͯΡ • ϳЄσ΅Blox frontendΨ᭗ͭͼREST APIͽ඙֢ͯΡ https://github.com/blox/blox/blob/dev/docs/frontend_design.md 47
  22. DynamoDB • v0.3.0;΅吖΀ΠϔЄόφϕί;ͭͼDynamoDBΨڥአͯΡ • Data ServiceአϓЄϣϸ • Environment;Deployment΄ᇫ䙪Ψכਂ • State

    ServiceአϓЄϣϸ • ECS΄ᇫ䙪Ψכਂ • Data ServiceአϓЄϣϸ΄DynamoDB Streams奺ኧͽScheduling ManagerΨ ޷ΉڊͭDeploymentΨ䋚ෞ 49
  23. Scheduling Manager • DeploymentΨ䋚ෞͭͼόφμΨπЀϓϗαЀφόЀφӤ΁ᯈᗝ • Step Functions͡Ο᩸㵕ͫ΢͵Lambdaͽ䋚ෞͯΡ • Ք஍Scheduler(Scheduling Controller)᮱ړ΅ڔΠڊͯԨਧ

    • Data ServiceΨ޷ΉڊͭͼEnvironment/Deployment΄ᇫ䙪Ψݐ ஑/כਂͯΡ • State ServiceΨ޷ΉڊͭͼECS΄ᇫ䙪Ψݐ஑/כਂͯΡ https://github.com/blox/blox/blob/dev/docs/daemon_design.md#scheduling-manager 50
  24. State Service • ECS EventsͽჁͫ΢͵ఘ䁭ΨDynamoDB΁כਂͯΡLambda͡Ο䯤౮ͫ΢Ρ • ECS΄ᇫ䙪΁䄜۸᩸ͣ͢͵䁰ݳ΁Ϸίϸόαϭͽ㳌ቘͯΡͩ;͢ݢᚆ • ݶ䦒΁Scheduled CloudWatch

    Events͡ΟInvokeͫ΢͵Lambda奺ኧͽμ϶φό΄ᇫ䙪Ψݐ஑ ͯΡΟ͚ͭ • ͵ͶECS Eventsͽ䱛ᚆ͢᪃ΠΡ΄͡䘁ΩͽΡ͹Γ͚ • ECS Events΅ͯΏͼ΄ఘ䁭Ψ㬃晄ͭͼͥ΢ΡΥͧͽ΅΀͚ • αϦЀϕΨݐΠͩΒͯݢᚆ௔͘͢Ρ • ϳЄσ΄ECS͡ΟAWSίθγЀϕ㯎(ϫϚЄυϖBlox͢䯤塈ͫ΢ΡίθγЀϕ)ΎαϦЀϕΨͿ ͜Κ͹ͼᭆΡ΄͡ https://github.com/blox/blox/blob/dev/docs/daemon_design.md#state-service 51
  25. daemon scheduler΄API API 㳌ቘ CreateEnvironment ෛ͚ͭEnvironmentΨ֢౮ StartDeployment ෛ͚ͭDeploymentΨ֢౮ RollbackDeployment զڹ΄Deployment΁䜟ͯ

    StopDeployment DeploymentΨ؊ྊͯΡ UpdateEnvironment EnvironmentΨๅෛ DeleteEnvironment EnvironmentΨڷᴻ GetEnvironment ᇙਧ΄EnvironmentΨᤒᐏ ListEnvironments قͼ΄EnvironmentΨᤒᐏ GetDeployment ᇙਧ΄DeploymentΨᤒᐏ ListDeployments قͼ΄DeploymentΨᤒᐏ https://github.com/blox/blox/blob/dev/docs/daemon_design.md#user-experience 52
  26. v1.0΄Environment΅ͩΩ΀ఽͮͽֵ͞ΡҘ # EnvironmentΨ֢౮ $ aws ecs create-environment \ --name <name>

    \ --task-definition <task-definition> \ --instance-group <instance-group> \ --role <role> \ --deployment-configuration <deployment-configuration> # قͼ΄EnvironmentΨᤒᐏ $ aws ecs list-environments \ --environment-type <environment-type> # ᇙਧ΄EnvironmentΨᤒᐏ $ aws ecs get-environment \ --environment-name <environment-name> https://github.com/blox/blox/blob/dev/docs/daemon_design.md#user-experience 53
  27. v1.0΄Deployment΅ͩΩ΀ఽͮͽֵ͞ΡҘ # DeploymentΨ֢౮ $ aws ecs start-deployment \ --environment-name <environment-name>

    \ --environment-version <environment-version> # قͼ΄DeploymentΨᤒᐏ $ aws ecs list-deployments \ --environment-name <environment-name> # ᇙਧ΄DeploymentΨᤒᐏ $ aws ecs get-deployment \ --environment-name <environment-name> \ --deployment-id <deployment-id> https://github.com/blox/blox/blob/dev/docs/daemon_design.md#user-experience 54