Gaming at Scale

In this presentation, I explain workload orchestration and how a good strategy can help you deal with regional- and provider-wide failures,.

This version of the talk was given at the 2023 Edition of Scale by the Bay in November 2023.

---

Companion Code: github.com/workloads

Transcript

2. Sr. Developer Advocate at HashiCorp he / him @ksatirli Kerim Satirli
3. Challenge: Build Collaboratively
4. conversational interface Laws of Robotics (sort of) observability
5. FROM !"platform=linux/amd64 node:18-alpine3.17 LABEL org.opencontainers.image.authors='ksatirli' LABEL org.opencontainers.image.url='github.com/workloads/minecraft-bot' LABEL org.opencontainers.image.description='Excavation Assistant' LABEL org.opencontainers.image.licenses='Apache-2.0' LABEL org.opencontainers.image.source='github.com/workloads/minecraft-bot' COPY ./ / WORKDIR / RUN npm install CMD [ "node", "dist/index.js" ] Dockerfile Container Image
6. AWS: us-west-2
7. job "minecraft_server" { datacenters = ["aws"] type = "service" namespace = "default" priority = 99 group "minecraft" { count = 1 ephemeral_disk { migrate = true size = 1024 sticky = true } network { mode = "host" Minecraft Server server.nomad.hcl
8. nomad run server.nomad.hcl Terminal Minecraft Server ==> Monitoring evaluation "e614d2f7" Evaluation triggered by job "minecraft_server" Allocation "82aa9f54" created Evaluation status: "complete" ==> Evaluation "e614d2f7" finished.
9. nomad-leader.svcs.dev:4646/ui/jobs Minecraft Server
10. nomad-leader.svcs.dev:4646/ui/jobs/minecraft_server@default Minecraft Server
11. Demo
12. nomad run bot.nomad.hcl Terminal Multi-Region Deployment ==> Monitoring evaluation "c6118496" Evaluation triggered by job "minecraft_bot" Allocation "8862fb0e" created Allocation "cae76d09" created Allocation "ff1f1972" created Evaluation status: "complete" ==> Evaluation "c6118496" finished.
13. nomad-leader.svcs.dev:4646/ui/jobs/minecraft_bot@default Minecraft Bot and Server
14. AWS: us-* AZ: *eu GCP: asia-* Multi-Cloud Deployment
15. aws_regions = toset([ "af-south-1", "ap-east-1", "ap-northeast-1", "ap-northeast-2", "ap-northeast-3", "ap-south-1", "ap-southeast-1", "ap-southeast-2", "ap-southeast-3", "ca-central-1", "eu-central-1", "eu-north-1", "eu-south-1", "eu-south-2", "eu-west-1", "eu-west-2", "eu-west-3", "me-south-1", "sa-east-1", Terraform Console Multi-Region Deployment
16. module "regional_aws_workspaces" { for_each = toset(data.aws_regions.main.names) source = "ksatirli/regional-workspace/tfe" version = "1.2.0" region = { category = "terraform" identifier = each.key prefix = "aws" variable = "aws_region" } vcs_repo = { identifier = "workloads/regional-aws-deployment" branch = "main" } } regional_deployments.tf Multi-Region Deployment
17. module "regional_aws_workspaces" { for_each = toset(data.aws_regions.main.names) source = "ksatirli/regional-workspace/tfe" version = "1.2.0" region = { category = "terraform" identifier = each.key prefix = "aws" variable = "aws_region" } vcs_repo = { identifier = "workloads/regional-aws-deployment" branch = "inactive" } } regional_deployments.tf Multi-Region Deployment
18. variable "cidr_map" { type = object({ geo = map(number) cardinality = map(number) }) description = "Mapping of CIDR data by AWS Region." default = { geo = { # Asia Central ac = 22 # Africa af = 23 # Asia Pacific ap = 27 Multi-Region Networking regional_cidrs.tf
19. app.terraform.io/app/workloads/workspaces Multi-Region Deployment
20. Demo
21. This talk wasn't about Minecraft.
22. This talk was about fault tolerance.
23. Take-aways ▪ train like you play - no manual ! ▪ codify infrastructure and processes ▪ architect for awareness, don't depend on it
24. Demo Code github.com/workloads/
25. speakerdeck.com/ksatirli