内部動作 - 削除時 25 Cloud Provider • AWS / GCP / Azure • LXD • OpenStack • etc etc ① Get a list of runner ③ Execute provider Virtual Machine with actions/runner ④ Delete an instance ② Check to delete runner
内部動作 - shoes-aws 29 Amazon Web Services Execute provider (hashicorp/go-plugin) EC with actions/runner shoes-aws EC with actions/runner EC with actions/runner Call AWS REST API .Runner Name to tag “Name” .Setup Script to cloud-init user-data .Resource Type to AWS Instance Type
内部動作 - shoes-lxd 30 LXD server Execute provider (hashicorp/go-plugin) LXD Container with actions/runner shoes-lxd Call LXD REST API LXD Container with actions/runner LXD Container with actions/runner .Runner Name to Instance name .Setup Script to cloud-init user-data .Resource Type to limits.cpu, limits.memory
Kubernetes Cluster CyberAgentにおけるmyshoes構成 38 myshoes-proxy (Global IP) github.com GHES User Tenant A Tenant B webhook myshoes REST API Stadium VM Stadium VM Stadium VMs LXD (listen ) Stadium VM Stadium VM Stadium VMs LXD (listen ) Stadium VM Stadium VM Stadium VMs LXD (listen ) k s Service k s Service for Service X for Service Y k s Service Tenant C k s Service
Kubernetes Cluster CyberAgentにおけるmyshoes構成 39 myshoes-proxy (Global IP) github.com GHES User Tenant A Tenant B webhook myshoes REST API Stadium VM Stadium VM Stadium VMs LXD (listen ) Stadium VM Stadium VM Stadium VMs LXD (listen ) Stadium VM Stadium VM Stadium VMs LXD (listen ) k s Service k s Service for Service X for Service Y k s Service Tenant C k s Service myshoes-proxy • HTTPプロキシ • webhook from GitHub • myshoes REST API • Prometheusメトリクス • Kubernetes Pod • svc.cluster.local 利⽤
Kubernetes Cluster CyberAgentにおけるmyshoes構成 40 myshoes-proxy (Global IP) github.com GHES User Tenant A Tenant B webhook myshoes REST API Stadium VM Stadium VM Stadium VMs LXD (listen ) Stadium VM Stadium VM Stadium VMs LXD (listen ) Stadium VM Stadium VM Stadium VMs LXD (listen ) k s Service k s Service for Service X for Service Y k s Service Tenant C k s Service myshoes-controller • operator-sdk • Tenant Resource • Service, Secret, Deployment
Kubernetes Cluster CyberAgentにおけるmyshoes構成 41 myshoes-proxy (Global IP) github.com GHES User Tenant A Tenant B webhook myshoes REST API Stadium VM Stadium VM Stadium VMs LXD (listen ) Stadium VM Stadium VM Stadium VMs LXD (listen ) Stadium VM Stadium VM Stadium VMs LXD (listen ) k s Service k s Service for Service X for Service Y k s Service Tenant C k s Service • shoes-lxd • job実⾏ごとに LXDインスタンス作成 • CoW + dockerdのためbtrfs • btrfs subvolume • NVMe volumeを直接利⽤
Kubernetes Cluster CyberAgentにおけるmyshoes構成 43 myshoes-proxy (Global IP) github.com GHES User Tenant A Tenant B webhook myshoes REST API Stadium VM Stadium VM Stadium VMs LXD (listen ) Stadium VM Stadium VM Stadium VMs LXD (listen ) Stadium VM Stadium VM Stadium VMs LXD (listen ) k s Service k s Service for Service X for Service Y k s Service Tenant C k s Service
CyberAgentにおけるmyshoes構成 44 Tenant A Tenant B Stadium VM Stadium VM Stadium VMs LXD (listen ) Stadium VM Stadium VM Stadium VMs LXD (listen ) Stadium VM Stadium VM Stadium VMs LXD (listen ) k s Service k s Service for Service X for Service Y k s Service Tenant C k s Service
既存 Cycloud CyberAgentにおけるmyshoes構成 45 Tenant A Tenant B Stadium VM Stadium VM Stadium VMs LXD (listen ) Stadium VM Stadium VM Stadium VMs LXD (listen ) Stadium VM Stadium VM Stadium VMs LXD (listen ) k s Service k s Service for Service X for Service Y k s Service Tenant C k s Service Service VM Service VM Service VM
既存 Cycloud CyberAgentにおけるmyshoes構成 46 Tenant A Tenant B Stadium VM Stadium VM Stadium VMs LXD (listen ) Stadium VM Stadium VM Stadium VMs LXD (listen ) Stadium VM Stadium VM Stadium VMs LXD (listen ) k s Service k s Service for Service X for Service Y k s Service Tenant C k s Service Service VM Service VM Service VM
既存 Cycloud CyberAgentにおけるmyshoes構成 47 Tenant A Tenant B Stadium VM Stadium VM Stadium VMs LXD (listen ) Stadium VM Stadium VM Stadium VMs LXD (listen ) Stadium VM Stadium VM Stadium VMs LXD (listen ) k s Service k s Service for Service X for Service Y k s Service Tenant C k s Service Service VM Service VM Service VM 既存 Cycloud ユーザは ⾃テナントのVMとの接続が可能 他テナントのVMとの接続は不可