Upgrade to Pro — share decks privately, control downloads, hide ads and more …

GitHub Actions runner基盤におけるオンプレミスマルチテナントアプリケーショ...

GitHub Actions runner基盤におけるオンプレミスマルチテナントアプリケーションの運用 #CADC2022

CyberAgent Developer Conference 2022 の登壇資料です https://cadc.cyberagent.co.jp/2022/program/github-actions-runner-oss-myshoes/

CIU (CyberAgent Infrastructure Unit) ではCycloudとよばれるプライベートクラウドを運用しており、2021年9月にGitHub Actiionsにおけるrunner基盤である Cycloud-hosted runner を提供開始しました。これは安価にかつ強力なGitHub Actions runnerを社内に向けて提供するものです。

GitHub-hostedと同様に動的なrunner作成を行い、既存のプライベートクラウド環境とハイブリッドな運用を行いながらもマルチテナントを提供するためにさまざまなレイヤーから取り組みを行っています。

本登壇では動的なrunner作成を行うOSSであるmyshoesについて紹介した上で、myshoesを利用したマネージドサービスCycloud-hosted runnerの技術仕様に触れ、なぜその仕様が必要だったのか、その仕様を実現するためにどのように取り組んだかについてお話します。

whywaita

March 24, 2022
Tweet

More Decks by whywaita

Other Decks in Technology

Transcript

  1. Virtual Machine with actions/runner Cloud Provider • AWS / GCP

    / Azure • LXD / OpenStack • etc etc... ① Start job ② webhook ③ CreateInstance ④ Create ⑤ Register
  2. $ZDMPVEIPTUFESVOOFS $ curl -XGET ${your_shoes_host}/target | jq . [ {

    "id": "477f6073-90d1-47d8-958f-4707cea61e8d" , "scope": "octocat" , "token_expired_at": "2006-01-02T15:04:05Z" , "ghe_domain": "https://github.example.com" , "resource_type": "micro" , "runner_user": "ubuntu" , "runner_version": "" , "provider_url": "" , "status": "active" , "status_description": "" , "created_at": "2006-01-02T15:04:05Z" , "updated_at": "2006-01-02T15:04:05Z " }
  3. $ZDMPVEIPTUFESVOOFS $ curl -XGET ${your_shoes_host}/target | jq . [ {

    "id": "477f6073-90d1-47d8-958f-4707cea61e8d" , "scope": "octocat", "token_expired_at": "2006-01-02T15:04:05Z" , "ghe_domain": "https://github.example.com" , "resource_type": "micro" , "runner_user": "ubuntu" , "runner_version": "" , "provider_url": "" , "status": "active" , "status_description": "" , "created_at": "2006-01-02T15:04:05Z" , "updated_at": "2006-01-02T15:04:05Z " } ಈ࡞͢Δ0SHPS3FQPࢦఆ
  4. $ZDMPVEIPTUFESVOOFS $ curl -XGET ${your_shoes_host}/target | jq . [ {

    "id": "477f6073-90d1-47d8-958f-4707cea61e8d" , "scope": "octocat" , "token_expired_at": "2006-01-02T15:04:05Z" , "ghe_domain": "https://github.example.com", "resource_type": "micro" , "runner_user": "ubuntu" , "runner_version": "" , "provider_url": "" , "status": "active" , "status_description": "" , "created_at": "2006-01-02T15:04:05Z" , "updated_at": "2006-01-02T15:04:05Z " } (JU)VC&OUFSQSJTF4FSWFSରԠ
  5. $ZDMPVEIPTUFESVOOFS $ curl -XGET ${your_shoes_host}/target | jq . [ {

    "id": "477f6073-90d1-47d8-958f-4707cea61e8d" , "scope": "octocat" , "token_expired_at": "2006-01-02T15:04:05Z" , "ghe_domain": "https://github.example.com" , "resource_type": "micro", "runner_user": "ubuntu" , "runner_version": "" , "provider_url": "" , "status": "active" , "status_description": "" , "created_at": "2006-01-02T15:04:05Z" , "updated_at": "2006-01-02T15:04:05Z " } ར༻͢ΔSVOOFSͷεϖοΫ 
 ˞0SHSFQP͝ͱʹมߋՄೳ
  6. $ZDMPVEIPTUFESVOOFS $ cat test.yaml
 jobs : test : - runs-on:

    ubuntu-lates t + runs-on: self-hoste d steps : - name: Checkout
  7. $ZDMPVEIPTUFESVOOFSߏ੒ Kubernetes Cluster myshoes-proxy (Global IP) github.com GHES User Tenant

    A Tenant B webhook myshoes REST API Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service
  8. $ZDMPVEIPTUFESVOOFSߏ੒ 既存 Cycloud VM Tenant A Tenant B Stadium VM

    Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service Service VM Service VM Service VM 既存 Cycloud ユーザは 
 ⾃テナントのVMとの接続が可能 
 他テナントのVMとの接続は不可
  9. $ZDMPVEIPTUFESVOOFSߏ੒ Kubernetes Cluster myshoes-proxy (Global IP) github.com GHES User Tenant

    A Tenant B webhook myshoes REST API Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service
  10. $ZDMPVEIPTUFESVOOFSߏ੒ Kubernetes Cluster myshoes-proxy (Global IP) github.com GHES User Tenant

    A Tenant B webhook myshoes REST API Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service <શମڞ༗>NZTIPFTQSPYZ  *OUFSOFUˠ1SJWBUF$MPVE  ֤ςφϯτ΁ৼΓ෼͚  ೝূೝՄ
  11. $ZDMPVEIPTUFESVOOFSߏ੒ Kubernetes Cluster myshoes-proxy (Global IP) github.com GHES User Tenant

    A Tenant B webhook myshoes REST API Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service <શମڞ༗> 
 NZTIPFTDPOUSPMMFS  5FOBOU $3% ͔Βੜ੒  LT4FSWJDF  NZTIPFT1PE  OPEF@TFMFDUPSͰ 
 ىಈςφϯτΛ੾ସ
  12. NZTIPFT 044 ͷ࣮૷ํ਑ w/PO(PBMͷ໌֬ԽϝϦοτ wʮͲ͜Ͱ࣮૷͢΂͖͔ʯ͕໌֬Խ wࣾ಺ཁ݅͸ଞ ϓϩΩγ ʹ೚ͤΔ wϚϧνςφϯτ wೝূೝՄ

    wޙ͔Βඞͣ௥Ճཁ݅͸΍ͬͯ͘Δ wlͲͷϚΠΫϩαʔϏεʹ௥Ճ͢Δ͔z 
 Λߟྀ͢Δඞཁͷͳ͍ઃܭ͕ॏཁ
  13. $ZDMPVEIPTUFESVOOFSߏ੒ Kubernetes Cluster myshoes-proxy (Global IP) github.com GHES User Tenant

    A Tenant B webhook myshoes REST API Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service
  14. $ZDMPVEIPTUFESVOOFSߏ੒ Kubernetes Cluster myshoes-proxy LXD (listen 8 44 3 )

    Stadium VM Stadium VM Stadium VMs 8 s Service k 8 s Service for Service X for Service Y k 8
  15. Virtual Machine with actions/runner Cloud Provider • AWS / GCP

    / Azure • LXD / OpenStack • etc etc... ① Start job ② webhook ③ CreateInstance ④ Create ⑤ Register
  16. Virtual Machine with actions/runner Cloud Provider • AWS / GCP

    / Azure • LXD / OpenStack • etc etc... ① Start job ② webhook ③ CreateInstance ④ Create ⑤ Register
  17. ಺෦ಈ࡞NZTIPFTQSPWJEFS Cloud Provider Execute provider (hashicorp/go-plugin) Virtual Machine with actions/runner

    shoes-xxx Virtual Machine with actions/runner Virtual Machine with actions/runner Call API 1 .Runner Name : runnerに付けるべき名前 2 .Setup Script : actions/runner展開スクリプト 3 .Resource Type : runnerのスペック
  18. ಺෦ಈ࡞TIPFTMYE Execute provider (hashicorp/go-plugin) shoes-lxd Call API Stadium lxd Virtual

    Machine with actions/runner Stadium lxd Virtual Machine with actions/runner ɾ 
 ɾ ɾ
  19. ಺෦ಈ࡞TIPFTMYE Execute provider (hashicorp/go-plugin) shoes-lxd Call API Stadium lxd Virtual

    Machine with actions/runner Stadium lxd Virtual Machine with actions/runner ɾ 
 ɾ ɾ rand.Intn
  20. ಺෦ಈ࡞TIPFTMYE Execute provider (hashicorp/go-plugin) shoes-lxd Call API Stadium lxd Virtual

    Machine with actions/runner Stadium lxd Virtual Machine with actions/runner ɾ 
 ɾ ɾ
  21. ಺෦ಈ࡞TIPFTMYENVMUJ Execute provider (hashicorp/go-plugin) shoes-lxd-multi Call API Stadium lxd Virtual

    Machine with actions/runner Stadium lxd Virtual Machine with actions/runner ɾ 
 ɾ ɾ shoes-lxd- server Call (gRPC)
  22. ಺෦ಈ࡞TIPFTMYENVMUJ Execute provider (hashicorp/go-plugin) shoes-lxd-multi Call API Stadium lxd Virtual

    Machine with actions/runner Stadium lxd Virtual Machine with actions/runner ɾ 
 ɾ ɾ shoes-lxd- server Call (gRPC)
  23. ܧଓతͳ#VJME6Q apiVersion: myshoes.managed.cycloud.io/v 1 kind: Tenant
 spec : domain: XX

    X project: YY Y plugin_url: https://github.com/whywaita/shoes-lxd-multi/ releases/download/v0.0.5/shoes-lxd-multi-linux-amd6 4 lxd_image_alias: (snip ) myshoes_version: v1.10.7
 environment_values : MAX_CONCURRENCY_DELETING: '2 ' github_apps: (snip)
 lxd_hosts : - host: https://(snip):8443 # myshoes-stadium-tenantA-00 1 - host: https://(snip):8443 # myshoes-stadium-tenantB-002
  24. ܧଓతͳ#VJME6Q apiVersion: myshoes.managed.cycloud.io/v 1 kind: Tenant
 spec : domain: XX

    X project: YY Y plugin_url: https://github.com/whywaita/shoes-lxd-multi/ releases/download/v0.0.5/shoes-lxd-multi-linux-amd6 4 lxd_image_alias: (snip ) myshoes_version: v1.10.7
 environment_values : MAX_CONCURRENCY_DELETING: '2 ' github_apps: (snip)
 lxd_hosts : - host: https://(snip):8443 # myshoes-stadium-tenantA-00 1 - host: https://(snip):8443 # myshoes-stadium-tenantB-002
  25. ܧଓతͳ#VJME6Q apiVersion: myshoes.managed.cycloud.io/v 1 kind: Tenant
 spec : domain: XX

    X project: YY Y plugin_url: http://10.X.Y.Z/shoes-lxd-multi-linux-amd6 4 lxd_image_alias: (snip ) myshoes_version: v1.10.7
 environment_values : MAX_CONCURRENCY_DELETING: '2 ' github_apps: (snip)
 lxd_hosts : - host: https://(snip):8443 # myshoes-stadium-tenantA-00 1 - host: https://(snip):8443 # myshoes-stadium-tenantB-002
  26. ܧଓతͳ#VJME6Q apiVersion: myshoes.managed.cycloud.io/v 1 kind: Tenant
 spec : domain: XX

    X project: YY Y plugin_url: https://github.com/whywaita/shoes-lxd-multi/ releases/download/v0.0.5/shoes-lxd-multi-linux-amd6 4 lxd_image_alias: (snip ) myshoes_version: v1.10.7
 environment_values : MAX_CONCURRENCY_DELETING: '2 ' github_apps: (snip)
 lxd_hosts : - host: https://(snip):8443 # myshoes-stadium-tenantA-00 1 - host: https://(snip):8443 # myshoes-stadium-tenantB-002
  27. ܧଓతͳ#VJME6Q apiVersion: myshoes.managed.cycloud.io/v 1 kind: Tenant
 spec : domain: XX

    X project: YY Y plugin_url: https://github.com/whywaita/shoes-lxd-multi/ releases/download/v0.0.5/shoes-lxd-multi-linux-amd6 4 lxd_image_alias: (snip ) myshoes_version: sha-xxxxxx
 environment_values : MAX_CONCURRENCY_DELETING: '2 ' github_apps: (snip)
 lxd_hosts : - host: https://(snip):8443 # myshoes-stadium-tenantA-00 1 - host: https://(snip):8443 # myshoes-stadium-tenantB-002
  28. ܧଓతͳ#VJME6Q apiVersion: myshoes.managed.cycloud.io/v 1 kind: Tenant
 spec : domain: XX

    X project: YY Y plugin_url: https://github.com/whywaita/shoes-lxd-multi/ releases/download/v0.0.5/shoes-lxd-multi-linux-amd6 4 lxd_image_alias: (snip ) myshoes_version: v1.10.7
 environment_values : MAX_CONCURRENCY_DELETING: '2 ' github_apps: (snip)
 lxd_hosts : - host: https://(snip):8443 # myshoes-stadium-tenantA-00 1 - host: https://(snip):8443 # myshoes-stadium-tenantB-002