Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Industrial Mining at Scale

Industrial Mining at Scale

In this presentation, I explain workload orchestration concepts and how they relate to spending more time doing fun things in Minecraft.

This version of the talk was given at the 2023 Edition of All Things Open in October 2023.

---

Companion Code: github.com/workloads

Kerim Satirli

October 17, 2023
Tweet

More Decks by Kerim Satirli

Other Decks in Technology

Transcript

  1. Industrial Mining at Scale
    (A Minecraft Story)

    View full-size slide

  2. Industrial Mining at Scale
    (A Minecraft Story)

    View full-size slide

  3. Sr. Developer Advocate at HashiCorp
    he / him
    @ksatirli
    Kerim
    Satirli

    View full-size slide

  4. Challenge:
    Dig less, build more.

    View full-size slide

  5. Let's build a
    Minecraft Bot

    View full-size slide

  6. chat interface and API
    (2 of 3) Laws of Robotics
    health endpoint

    View full-size slide

  7. FROM !"platform=linux/amd64 node:18-alpine3.16
    LABEL org.opencontainers.image.authors='ksatirli'
    LABEL org.opencontainers.image.url='github.com/workloads/minecraft-bot'
    LABEL org.opencontainers.image.description='Excavation Assistant'
    LABEL org.opencontainers.image.licenses='Apache-2.0'
    LABEL org.opencontainers.image.source='github.com/workloads/minecraft-bot'
    COPY ./ /
    WORKDIR /
    RUN npm install
    CMD [ "node", "dist/index.js" ]
    Dockerfile
    Container Image

    View full-size slide

  8. AWS: us-west-2

    View full-size slide

  9. job "minecraft" {
    datacenters = ["aws"]
    type = "service"
    namespace = "default"
    priority = 99
    group "minecraft" {
    count = 1
    ephemeral_disk {
    migrate = true
    size = 1024
    sticky = true
    }
    network {
    mode = "host"
    Minecraft Server
    server.nomad.hcl

    View full-size slide

  10. nomad run server.nomad.hcl
    Terminal
    Minecraft Server
    ==> Monitoring evaluation "e614d2f7"
    Evaluation triggered by job "minecraft"
    Allocation "82aa9f54" created
    Evaluation status: "complete"
    ==> Evaluation "e614d2f7" finished.

    View full-size slide

  11. nomad-leader.svcs.dev:4646
    Minecraft Server

    View full-size slide

  12. Let's deploy a
    Minecraft Bot

    View full-size slide

  13. job "minecraft-bot" {
    task "minecraft_bot" {
    driver = "docker"
    config {
    image = "ghcr.io/workloads/minecraft-bot:v20231010"
    ports = [
    "api",
    "inventory",
    ]
    }
    env {
    # APP CONFIGURATION
    DEBUG = "false"
    LOG_COLOR = "true"
    Minecraft Bot
    bot.nomad.hcl

    View full-size slide

  14. nomad-leader.svcs.dev:4646
    Minecraft Bot

    View full-size slide

  15. nomad run bot.nomad.hcl
    Terminal
    Multi-Region Deployment
    ==> Monitoring evaluation "c6118496"
    Evaluation triggered by job "bot"
    Allocation "8862fb0e" created
    Allocation "cae76d09" created
    Allocation "ff1f1972" created
    Evaluation status: "complete"
    ==> Evaluation "c6118496" finished.

    View full-size slide

  16. EMEA
    AMER APAC
    Multi-Region Deployment

    View full-size slide

  17. data "aws_regions" "main" {}
    regional_deployments.tf
    Multi-Region Deployment

    View full-size slide

  18. aws_regions = toset([
    "af-south-1",
    "ap-east-1",
    "ap-northeast-1",
    "ap-northeast-2",
    "ap-northeast-3",
    "ap-south-1",
    "ap-southeast-1",
    "ap-southeast-2",
    "ap-southeast-3",
    "ca-central-1",
    "eu-central-1",
    "eu-north-1",
    "eu-south-1",
    "eu-south-2",
    "eu-west-1",
    "eu-west-2",
    "eu-west-3",
    "me-south-1",
    "sa-east-1",
    Terraform Console
    Multi-Region Deployment

    View full-size slide

  19. module "regional_aws_workspaces" {
    for_each = toset(data.aws_regions.main.names)
    source = "ksatirli/regional-workspace/tfe"
    version = "1.2.0"
    region = {
    category = "terraform"
    identifier = each.key
    prefix = "aws"
    variable = "aws_region"
    }
    vcs_repo = {
    identifier = "workloads/regional-aws-deployment"
    branch = "main"
    }
    }
    regional_deployments.tf
    Multi-Region Deployment

    View full-size slide

  20. Multi-Region Deployment

    View full-size slide

  21. console.aws.amazon.com/support
    Multi-Region Deployment

    View full-size slide

  22. ✉ from AWS Support

    View full-size slide

  23. app.terraform.io/app/workloads/workspaces
    Multi-Region Deployment

    View full-size slide

  24. deployed an army of bots to do our bidding
    learned about Cloud Geography
    provisioned resilient infrastructure
    built a bot and containerized it
    Recap

    View full-size slide

  25. Demo Code
    github.com/workloads/

    View full-size slide

  26. speakerdeck.com/ksatirli

    View full-size slide