Share
Embed
Development myshoes and Provide Cycloud-hosted runner -- GitHub Actions with your shoes. #cndjp
Transcript
62 slides
-
GitHub Actions with your shoes. Development myshoes and Provide
Cycloud-hosted runner CyberAgent, Inc. CyberAgent group Infrastructure Unit Private Cloud Team Cloud Maker / Nakanishi Kento @whywaita Cloud Native Developers # 17 僕たち私たちの CI/CD はこれだ! @ ONLINE 2021 / 1 0 / 0 8 1 -
whywaita • Nakanishi Kento • CloudMaker in CyberAgent since 2
0 19 CIU Private Cloud team • Work: Dev and Ops using OpenStack / Kubernetes • Hobby: homelab and Anime-song • Become Founder of AS 63 7 9 0 (CA-LAB) 🥰 2 -
1 .Runner in GitHub Actions 2 .What’s myshoes 3 .myshoes
in CyberAgent 3 -
GitHub Actions runner • GitHub-hosted runner • Managed by GitHub
• Backend as a Azure Pipelines • self-hosted runner • Managed by you • github.com/actions/runner install to own server • On-prem, Cloud, homelab … 5 -
GitHub-hosted runner • Pros • Fully managed • Cons •
Fixed spec (CPU: 2 , Mem: 7 GB) • Need many money 💸 6 -
self-hosted runner • Pros • Running own cloud, infrastructure •
Using small cost and own IP address • Cons • Not Consistent (actions/runner is daemon) • “running terraform that managed self-hosted runner in self- hosted runner … ” 7 -
whywaita/myshoes • Auto scaling self-hosted runner for GitHub Actions •
Creating self-hosted runner per job • Available your infrastructure • shoes-aws, shoes-lxd, and you can implement • https://github.com/whywaita/myshoes 12 -
Creating per job, Can use consistent 19 Creating a new
runner per GitHub Actions work fl ows -
Creating per job, Can use consistent Runner Creating a new
runner per GitHub Actions work fl ows -
Creating per job, Can use consistent Runner Runner Creating a
new runner per GitHub Actions work fl ows -
Creating per job, Can use consistent Runner Runner Runner Runner
Runner Runner Creating a new runner per GitHub Actions work fl ows -
Support matrix 26 Runner Runner Creating a new runner per
GitHub Actions jobs -
Internal - Add runner 27 Cloud Provider • AWS /
GCP / Azure • LXD • OpenStack • etc etc … ② Send a webhook to myshoes ① Request job ③ Execute provider Virtual Machine with actions/runner ④ Create an instance ⑤ Register runner -
Internal - Delete runner 28 Cloud Provider • AWS /
GCP / Azure • LXD • OpenStack • etc etc … ① Get a list of runner ③ Execute provider Virtual Machine with actions/runner ④ Delete an instance ② Check to delete runner -
Internal - Add runner 29 Cloud Provider • AWS /
GCP / Azure • LXD • OpenStack • etc etc … ② Send a webhook to myshoes ① Request job ③ Execute provider Virtual Machine with actions/runner ④ Create an instance ⑤ Register runner -
Internal - Add runner 30 Cloud Provider • AWS /
GCP / Azure • LXD • OpenStack • etc etc … ③ Execute provider Virtual Machine with actions/runner ④ Create an instance ② Send a webhook -
Internal - shoes-provider 31 Cloud Provider Execute provider (hashicorp/go-plugin) Virtual
Machine with actions/runner shoes-xxx Virtual Machine with actions/runner Virtual Machine with actions/runner Call API 1 .Runner Name : Name of runner 2 .Setup Script : Script of provisioning actions/runner 3 .Resource Type : Spec for runner -
shoes-provider example: shoes-aws 32 Amazon Web Services Execute provider (hashicorp/go-plugin)
EC 2 with actions/runner shoes-aws EC 2 with actions/runner EC 2 with actions/runner Call AWS REST API 1 .Runner Name to tag “Name” 2 .Setup Script to cloud-init user-data 3 .Resource Type to AWS Instance Type -
shoes-provider example: shoes-lxd 33 LXD server Execute provider (hashicorp/go-plugin) LXD
Container with actions/runner shoes-lxd Call LXD REST API LXD Container with actions/runner LXD Container with actions/runner 1 .Runner Name to Instance name 2 .Setup Script to cloud-init user-data 3 .Resource Type to limits.cpu, limits.memory -
myshoes-provider has dream 🤩 • Cheap instance in Public cloud
• AWS Spot Instances, GCP Preemptible Instances • Private Cloud, homelab, On-prem • LXD, OpenStack • Speci fi c hardwares • GPU, FPGA 34 -
Cycloud • Cy(berAgent)cloud • Private Cloud • IaaS, Managed Service
• Customized for CyberAgent products • CA products can use Compute at a low cost 💰 36 -
CI in CyberAgent • Products decide to use technology, CI/CD
• github.com , GitHub Enterprise Server, GitLab • Jenkins, Travis CI, CircleCI, Drone CI, Bitrise • In many case, use enterprise application ex:) CircleCI Enterprise • A few products start to use GitHub Actions in beta 37 -
CI in CyberAgent • Need to migrate Circle CI Enterprise
to GitHub Actions • End of using Enterprise • Bye cost merit • Increase support cost • whywaita “We can get a cost down if use Cycloud?” → Start to develop myshoes … 38 -
myshoes in CyberAgent • Cycloud-hosted runner has released 🥳 •
Managed service myshoes • Support GHES, GHEC, github.com • Cover many use cases in CyberAgent • Multi-tenancy 41 -
myshoes in CyberAgent • whywaita/myshoes + shoes-lxd • multi-tenancy: myshoes-controller
• Implemented Kubernetes Controller • MySQL Cluster (with orchestrator) • Docker Registry v 2 , LXD image server 42 -
Kubernetes Cluster Architecture of Cycloud-hosted runner 43 myshoes-proxy (Global IP)
github.com GHES User Tenant A Tenant B webhook myshoes REST API Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service -
Kubernetes Cluster Architecture of Cycloud-hosted runner 44 myshoes-proxy (Global IP)
github.com GHES User Tenant A Tenant B webhook myshoes REST API Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service myshoes-proxy • HTTP Proxy • webhook from GitHub • myshoes REST API • Use svc.cluster.local • Prometheus metrics • Usage API for billing -
Kubernetes Cluster Architecture of Cycloud-hosted runner 45 myshoes-proxy (Global IP)
github.com GHES User Tenant A Tenant B webhook myshoes REST API Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service • shoes-lxd • shoes-provider for LXD • btrfs (CoW + dockerd) • btrfs subvolume • Mount NVMe volume -
Kubernetes Cluster Architecture of Cycloud-hosted runner 47 myshoes-proxy (Global IP)
github.com GHES User Tenant A Tenant B webhook myshoes REST API Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service -
Architecture of Cycloud-hosted runner 48 Tenant A Tenant B Stadium
VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service -
Cycloud Architecture of Cycloud-hosted runner 49 Tenant A Tenant B
Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service Service VM Service VM Service VM -
Cycloud Architecture of Cycloud-hosted runner 50 Tenant A Tenant B
Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service Service VM Service VM Service VM -
Cycloud Architecture of Cycloud-hosted runner 51 Tenant A Tenant B
Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service Service VM Service VM Service VM Cycloud user is … Connected to my tenant is OK Anything complete in Cycloud! -
Spec of Cycloud-hosted runner • Stadium as a Cycloud Virtual
Machine • We plan to provide cost less than GitHub-hosted • runner image is almost the same as GitHub-hosted • https://github.com/whywaita/virtual-environments-lxd • + con fi gure apt repository, etc … • Spec of runner: 2 Core/ 20 GB ~ 1 6 Core/ 160 GB 52 -
Feature works • For OSS • more shoes provider, more
Documents • Support `--ephemeral` • For Cycloud-hosted runner • GPU runner ? • macOS runner? 53 -
Conclusion • Explain about runner for GitHub Actions • Pros
/ Cons • whywaita/myshoes is Good solution ☺ • It can manage self-hosted runner usefully! • We will develop myshoes and provide Cycloud-hosted runner • Please user report, Issue, Pull-Request ! 54 -
Appendix 1 : myshoes vs • actions-runner-controller/actions-runner-controller (top of star
in awesome- runners) • runner as a Kubernetes Pod • philips-labs/terraform-aws-github-runner, github-developer/self-hosted- runners-anthos • runner as a Public cloud instances or managed service • whywaita/myshoes • runner as a selective (able to implement shoes-providers) • more information: https://jonico.github.io/awesome-runners/ 55 -
Appendix 2 : customize myshoes • myshoes has a few
components • Implemented `interface()` in Go • Modify cmd.go (as main.go), Get customized myshoes ! 56 -
Appendix 2 : customize myshoes 57 https://github.com/whywaita/myshoes/blob/ 5 d 2
2 108 c 2 d 7 d 50 b 83 f 9 7 993 a 05 69 b 8 860 48 d 1 c 7 9 /cmd/server/cmd.go -
Appendix 2 : customize myshoes • myshoes components • datastore:
Persistent datastore (MySQL) • safety: Check can creating (Unlimited) • runner manager: Delete old runners 58 -
Appendix 2 : customize myshoes 59 https://github.com/whywaita/myshoes/blob/ 5 d 221
08 c 2 d 7 d 50 b 83 f 97993 a 0 569 b 88 60 48 d 1 c 79 /pkg/starter/safety/safety.go -
Appendix 2 : customize myshoes 60 https://github.com/whywaita/myshoes/blob/ 5 d 221
08 c 2 d 7 d 50 b 83 f 97993 a 0 569 b 88 60 48 d 1 c 79 /pkg/starter/starter.go -
Appendix 2 : customize myshoes 61 https://github.com/whywaita/myshoes/blob/ 5 d 221
08 c 2 d 7 d 50 b 83 f 97993 a 0 569 b 88 60 48 d 1 c 79 /pkg/starter/safety/unlimited/unlimited.go -
Appendix 2 : customize myshoes • Switch other Safety() implements
• Reason of waiting … • For billing • For stadium resource • For Legal (like HTTP 45 1 ) 62