Development myshoes and Provide Cycloud-hosted runner -- GitHub Actions with your shoes. #cndjp

Talked by https://cnd.connpass.com/event/223825/

Original: https://www.slideshare.net/whywaita/development-myshoes-and-provide-cycloudhosted-runner-github-actions-with-your-shoes

This slide is re-upload by the author.

Transcript

1. GitHub Actions with your shoes. Development myshoes and 
 Provide Cycloud-hosted runner CyberAgent, Inc. 
 CyberAgent group Infrastructure Unit Private Cloud Team 
 Cloud Maker / Nakanishi Kento @whywaita Cloud Native Developers # 17 僕たち私たちの CI/CD はこれだ！ @ ONLINE 
 2021 / 1 0 / 0 8 1
2. whywaita • Nakanishi Kento • CloudMaker in CyberAgent since 2 0 19 
 CIU Private Cloud team • Work: Dev and Ops using OpenStack / Kubernetes • Hobby: homelab and Anime-song • Become Founder of AS 63 7 9 0 (CA-LAB) 🥰 2
3. 1 .Runner in GitHub Actions 2 .What’s myshoes 3 .myshoes in CyberAgent 3
4. Runner in GitHub Actions 4
5. GitHub Actions runner • GitHub-hosted runner • Managed by GitHub • Backend as a Azure Pipelines • self-hosted runner • Managed by you • github.com/actions/runner install to own server • On-prem, Cloud, homelab … 5
6. GitHub-hosted runner • Pros • Fully managed • Cons • Fixed spec (CPU: 2 , Mem: 7 GB) • Need many money 💸 6
7. self-hosted runner • Pros • Running own cloud, infrastructure • Using small cost and own IP address • Cons • Not Consistent (actions/runner is daemon) • “running terraform that managed self-hosted runner in self- hosted runner … ” 7
8. 😩😩😩 8
9. Our solution is … 9
11. 11 What’s
12. whywaita/myshoes • Auto scaling self-hosted runner for GitHub Actions • Creating self-hosted runner per job • Available your infrastructure • shoes-aws, shoes-lxd, and you can implement • https://github.com/whywaita/myshoes 12
13. Usage - Execute job 13 Execute job …
14. Usage - Creating runner 14 Auto add by myshoes
15. Usage - End job 15 End job …
16. Usage - Deleting runner 16 Auto delete by myshoes
17. Can use as a GitHub-hosted 😍 17
18. Creating per job, Can use consistent 18
19. Creating per job, Can use consistent 19 Creating a new runner per GitHub Actions work fl ows
20. Creating per job, Can use consistent Runner Creating a new runner per GitHub Actions work fl ows
21. Creating per job, Can use consistent Runner Runner Creating a new runner per GitHub Actions work fl ows
22. Creating per job, Can use consistent Runner Runner Runner Runner Runner Runner Creating a new runner per GitHub Actions work fl ows
23. Support matrix 23
24. Support matrix 24
25. Support matrix 25
26. Support matrix 26 Runner Runner Creating a new runner per GitHub Actions jobs
27. Internal - Add runner 27 Cloud Provider • AWS / GCP / Azure • LXD • OpenStack • etc etc … ② Send a webhook 
   to myshoes ① Request job ③ Execute provider Virtual Machine with actions/runner ④ Create an instance ⑤ Register runner
28. Internal - Delete runner 28 Cloud Provider • AWS / GCP / Azure • LXD • OpenStack • etc etc … ① Get a list of runner ③ Execute provider Virtual Machine with actions/runner ④ Delete an instance ② Check to delete runner
29. Internal - Add runner 29 Cloud Provider • AWS / GCP / Azure • LXD • OpenStack • etc etc … ② Send a webhook 
   to myshoes ① Request job ③ Execute provider Virtual Machine with actions/runner ④ Create an instance ⑤ Register runner
30. Internal - Add runner 30 Cloud Provider • AWS / GCP / Azure • LXD • OpenStack • etc etc … ③ Execute provider Virtual Machine with actions/runner ④ Create an instance ② Send a webhook 

31. Internal - shoes-provider 31 Cloud Provider Execute provider (hashicorp/go-plugin) Virtual Machine with actions/runner shoes-xxx Virtual Machine with actions/runner Virtual Machine with actions/runner Call API 1 .Runner Name : Name of runner 2 .Setup Script : Script of provisioning actions/runner 3 .Resource Type : Spec for runner
32. shoes-provider example: shoes-aws 32 Amazon Web Services Execute provider (hashicorp/go-plugin) EC 2 with actions/runner shoes-aws EC 2 with actions/runner EC 2 with actions/runner Call AWS REST API 1 .Runner Name to tag “Name” 2 .Setup Script to cloud-init user-data 3 .Resource Type to AWS Instance Type
33. shoes-provider example: shoes-lxd 33 LXD server Execute provider (hashicorp/go-plugin) LXD Container with actions/runner shoes-lxd Call LXD REST API LXD Container with actions/runner LXD Container with actions/runner 1 .Runner Name to Instance name 2 .Setup Script to cloud-init user-data 3 .Resource Type to limits.cpu, limits.memory
34. myshoes-provider has dream 🤩 • Cheap instance in Public cloud • AWS Spot Instances, GCP Preemptible Instances • Private Cloud, homelab, On-prem • LXD, OpenStack • Speci fi c hardwares • GPU, FPGA 34
35. myshoes in CyberAgent 35
36. Cycloud • Cy(berAgent)cloud • Private Cloud • IaaS, Managed Service • Customized for CyberAgent products • CA products can use Compute at a low cost 💰 36
37. CI in CyberAgent • Products decide to use technology, CI/CD • github.com , GitHub Enterprise Server, GitLab • Jenkins, Travis CI, CircleCI, Drone CI, Bitrise • In many case, use enterprise application 
 ex:) CircleCI Enterprise • A few products start to use GitHub Actions in beta 37
38. CI in CyberAgent • Need to migrate Circle CI Enterprise to GitHub Actions • End of using Enterprise • Bye cost merit • Increase support cost • whywaita “We can get a cost down if use Cycloud?” → Start to develop myshoes … 38
39. myshoes in CyberAgent 39
40. myshoes in CyberAgent ( 20 2 1 / 0 9 ~) 40
41. myshoes in CyberAgent • Cycloud-hosted runner has released 🥳 • Managed service myshoes • Support GHES, GHEC, github.com • Cover many use cases in CyberAgent • Multi-tenancy 41
42. myshoes in CyberAgent • whywaita/myshoes + shoes-lxd • multi-tenancy: myshoes-controller • Implemented Kubernetes Controller • MySQL Cluster (with orchestrator) • Docker Registry v 2 , LXD image server 42
43. Kubernetes Cluster Architecture of Cycloud-hosted runner 43 myshoes-proxy (Global IP) github.com GHES User Tenant A Tenant B webhook myshoes REST API Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service
44. Kubernetes Cluster Architecture of Cycloud-hosted runner 44 myshoes-proxy (Global IP) github.com GHES User Tenant A Tenant B webhook myshoes REST API Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service myshoes-proxy • HTTP Proxy • webhook from GitHub • myshoes REST API • Use svc.cluster.local • Prometheus metrics • Usage API for billing
45. Kubernetes Cluster Architecture of Cycloud-hosted runner 45 myshoes-proxy (Global IP) github.com GHES User Tenant A Tenant B webhook myshoes REST API Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service • shoes-lxd • shoes-provider for LXD • btrfs (CoW + dockerd) • btrfs subvolume • Mount NVMe volume
46. And more … 46
47. Kubernetes Cluster Architecture of Cycloud-hosted runner 47 myshoes-proxy (Global IP) github.com GHES User Tenant A Tenant B webhook myshoes REST API Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service
48. Architecture of Cycloud-hosted runner 48 Tenant A Tenant B Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service
49. Cycloud Architecture of Cycloud-hosted runner 49 Tenant A Tenant B Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service Service VM Service VM Service VM
50. Cycloud Architecture of Cycloud-hosted runner 50 Tenant A Tenant B Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service Service VM Service VM Service VM
51. Cycloud Architecture of Cycloud-hosted runner 51 Tenant A Tenant B Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service Service VM Service VM Service VM Cycloud user is …
 Connected to my tenant is OK Anything complete in Cycloud!
52. Spec of Cycloud-hosted runner • Stadium as a Cycloud Virtual Machine • We plan to provide cost less than GitHub-hosted • runner image is almost the same as GitHub-hosted • https://github.com/whywaita/virtual-environments-lxd • + con fi gure apt repository, etc … • Spec of runner: 2 Core/ 20 GB ~ 1 6 Core/ 160 GB 52
53. Feature works • For OSS • more shoes provider, more Documents • Support `--ephemeral` • For Cycloud-hosted runner • GPU runner ? • macOS runner? 53
54. Conclusion • Explain about runner for GitHub Actions • Pros / Cons • whywaita/myshoes is Good solution ☺ • It can manage self-hosted runner usefully! • We will develop myshoes and 
 provide Cycloud-hosted runner • Please user report, Issue, Pull-Request ! 54
55. Appendix 1 : myshoes vs • actions-runner-controller/actions-runner-controller (top of star in awesome- runners) • runner as a Kubernetes Pod • philips-labs/terraform-aws-github-runner, github-developer/self-hosted- runners-anthos • runner as a Public cloud instances or managed service • whywaita/myshoes • runner as a selective (able to implement shoes-providers) • more information: https://jonico.github.io/awesome-runners/ 55
56. Appendix 2 : customize myshoes • myshoes has a few components • Implemented `interface()` in Go • Modify cmd.go (as main.go), 
 Get customized myshoes ! 56
57. Appendix 2 : customize myshoes 57 https://github.com/whywaita/myshoes/blob/ 5 d 2 2 108 c 2 d 7 d 50 b 83 f 9 7 993 a 05 69 b 8 860 48 d 1 c 7 9 /cmd/server/cmd.go
58. Appendix 2 : customize myshoes • myshoes components • datastore: Persistent datastore (MySQL) • safety: Check can creating (Unlimited) • runner manager: Delete old runners 58
59. Appendix 2 : customize myshoes 59 https://github.com/whywaita/myshoes/blob/ 5 d 221 08 c 2 d 7 d 50 b 83 f 97993 a 0 569 b 88 60 48 d 1 c 79 /pkg/starter/safety/safety.go
60. Appendix 2 : customize myshoes 60 https://github.com/whywaita/myshoes/blob/ 5 d 221 08 c 2 d 7 d 50 b 83 f 97993 a 0 569 b 88 60 48 d 1 c 79 /pkg/starter/starter.go
61. Appendix 2 : customize myshoes 61 https://github.com/whywaita/myshoes/blob/ 5 d 221 08 c 2 d 7 d 50 b 83 f 97993 a 0 569 b 88 60 48 d 1 c 79 /pkg/starter/safety/unlimited/unlimited.go
62. Appendix 2 : customize myshoes • Switch other Safety() implements • Reason of waiting … • For billing • For stadium resource • For Legal (like HTTP 45 1 ) 62