Development myshoes and Provide Cycloud-hosted runner -- GitHub Actions with your shoes. #cndjp

Slide 1

Slide 1 text

GitHub Actions with your shoes. Development myshoes and   Provide Cycloud-hosted runner CyberAgent, Inc.   CyberAgent group Infrastructure Unit Private Cloud Team   Cloud Maker / Nakanishi Kento @whywaita Cloud Native Developers # 17 僕たち私たちの CI/CD はこれだ！ @ ONLINE   2021 / 1 0 / 0 8 1

Slide 2

Slide 2 text

whywaita • Nakanishi Kento • CloudMaker in CyberAgent since 2 0 19   CIU Private Cloud team • Work: Dev and Ops using OpenStack / Kubernetes • Hobby: homelab and Anime-song • Become Founder of AS 63 7 9 0 (CA-LAB) 🥰 2

Slide 3

Slide 3 text

1 .Runner in GitHub Actions 2 .What’s myshoes 3 .myshoes in CyberAgent 3

Slide 4

Slide 4 text

Runner in GitHub Actions 4

Slide 5

Slide 5 text

GitHub Actions runner • GitHub-hosted runner • Managed by GitHub • Backend as a Azure Pipelines • self-hosted runner • Managed by you • github.com/actions/runner install to own server • On-prem, Cloud, homelab … 5

Slide 6

Slide 6 text

GitHub-hosted runner • Pros • Fully managed • Cons • Fixed spec (CPU: 2 , Mem: 7 GB) • Need many money 💸 6

Slide 7

Slide 7 text

self-hosted runner • Pros • Running own cloud, infrastructure • Using small cost and own IP address • Cons • Not Consistent (actions/runner is daemon) • “running terraform that managed self-hosted runner in self- hosted runner … ” 7

Slide 8

Slide 8 text

😩😩😩 8

Slide 9

Slide 9 text

Our solution is … 9

Slide 10

Slide 10 text

Slide 11

Slide 11 text

11 What’s

Slide 12

Slide 12 text

whywaita/myshoes • Auto scaling self-hosted runner for GitHub Actions • Creating self-hosted runner per job • Available your infrastructure • shoes-aws, shoes-lxd, and you can implement • https://github.com/whywaita/myshoes 12

Slide 13

Slide 13 text

Usage - Execute job 13 Execute job …

Slide 14

Slide 14 text

Usage - Creating runner 14 Auto add by myshoes

Slide 15

Slide 15 text

Usage - End job 15 End job …

Slide 16

Slide 16 text

Usage - Deleting runner 16 Auto delete by myshoes

Slide 17

Slide 17 text

Can use as a GitHub-hosted 😍 17

Slide 18

Slide 18 text

Creating per job, Can use consistent 18

Slide 19

Slide 19 text

Creating per job, Can use consistent 19 Creating a new runner per GitHub Actions work fl ows

Slide 20

Slide 20 text

Creating per job, Can use consistent Runner Creating a new runner per GitHub Actions work fl ows

Slide 21

Slide 21 text

Creating per job, Can use consistent Runner Runner Creating a new runner per GitHub Actions work fl ows

Slide 22

Slide 22 text

Creating per job, Can use consistent Runner Runner Runner Runner Runner Runner Creating a new runner per GitHub Actions work fl ows

Slide 23

Slide 23 text

Support matrix 23

Slide 24

Slide 24 text

Support matrix 24

Slide 25

Slide 25 text

Support matrix 25

Slide 26

Slide 26 text

Support matrix 26 Runner Runner Creating a new runner per GitHub Actions jobs

Slide 27

Slide 27 text

Internal - Add runner 27 Cloud Provider • AWS / GCP / Azure • LXD • OpenStack • etc etc … ② Send a webhook   to myshoes ① Request job ③ Execute provider Virtual Machine with actions/runner ④ Create an instance ⑤ Register runner

Slide 28

Slide 28 text

Internal - Delete runner 28 Cloud Provider • AWS / GCP / Azure • LXD • OpenStack • etc etc … ① Get a list of runner ③ Execute provider Virtual Machine with actions/runner ④ Delete an instance ② Check to delete runner

Slide 29

Slide 29 text

Internal - Add runner 29 Cloud Provider • AWS / GCP / Azure • LXD • OpenStack • etc etc … ② Send a webhook   to myshoes ① Request job ③ Execute provider Virtual Machine with actions/runner ④ Create an instance ⑤ Register runner

Slide 30

Slide 30 text

Internal - Add runner 30 Cloud Provider • AWS / GCP / Azure • LXD • OpenStack • etc etc … ③ Execute provider Virtual Machine with actions/runner ④ Create an instance ② Send a webhook  

Slide 31

Slide 31 text

Internal - shoes-provider 31 Cloud Provider Execute provider (hashicorp/go-plugin) Virtual Machine with actions/runner shoes-xxx Virtual Machine with actions/runner Virtual Machine with actions/runner Call API 1 .Runner Name : Name of runner 2 .Setup Script : Script of provisioning actions/runner 3 .Resource Type : Spec for runner

Slide 32

Slide 32 text

shoes-provider example: shoes-aws 32 Amazon Web Services Execute provider (hashicorp/go-plugin) EC 2 with actions/runner shoes-aws EC 2 with actions/runner EC 2 with actions/runner Call AWS REST API 1 .Runner Name to tag “Name” 2 .Setup Script to cloud-init user-data 3 .Resource Type to AWS Instance Type

Slide 33

Slide 33 text

shoes-provider example: shoes-lxd 33 LXD server Execute provider (hashicorp/go-plugin) LXD Container with actions/runner shoes-lxd Call LXD REST API LXD Container with actions/runner LXD Container with actions/runner 1 .Runner Name to Instance name 2 .Setup Script to cloud-init user-data 3 .Resource Type to limits.cpu, limits.memory

Slide 34

Slide 34 text

myshoes-provider has dream 🤩 • Cheap instance in Public cloud • AWS Spot Instances, GCP Preemptible Instances • Private Cloud, homelab, On-prem • LXD, OpenStack • Speci fi c hardwares • GPU, FPGA 34

Slide 35

Slide 35 text

myshoes in CyberAgent 35

Slide 36

Slide 36 text

Cycloud • Cy(berAgent)cloud • Private Cloud • IaaS, Managed Service • Customized for CyberAgent products • CA products can use Compute at a low cost 💰 36

Slide 37

Slide 37 text

CI in CyberAgent • Products decide to use technology, CI/CD • github.com , GitHub Enterprise Server, GitLab • Jenkins, Travis CI, CircleCI, Drone CI, Bitrise • In many case, use enterprise application   ex:) CircleCI Enterprise • A few products start to use GitHub Actions in beta 37

Slide 38

Slide 38 text

CI in CyberAgent • Need to migrate Circle CI Enterprise to GitHub Actions • End of using Enterprise • Bye cost merit • Increase support cost • whywaita “We can get a cost down if use Cycloud?” → Start to develop myshoes … 38

Slide 39

Slide 39 text

myshoes in CyberAgent 39

Slide 40

Slide 40 text

myshoes in CyberAgent ( 20 2 1 / 0 9 ~) 40

Slide 41

Slide 41 text

myshoes in CyberAgent • Cycloud-hosted runner has released 🥳 • Managed service myshoes • Support GHES, GHEC, github.com • Cover many use cases in CyberAgent • Multi-tenancy 41

Slide 42

Slide 42 text

myshoes in CyberAgent • whywaita/myshoes + shoes-lxd • multi-tenancy: myshoes-controller • Implemented Kubernetes Controller • MySQL Cluster (with orchestrator) • Docker Registry v 2 , LXD image server 42

Slide 43

Slide 43 text

Kubernetes Cluster Architecture of Cycloud-hosted runner 43 myshoes-proxy (Global IP) github.com GHES User Tenant A Tenant B webhook myshoes REST API Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service

Slide 44

Slide 44 text

Kubernetes Cluster Architecture of Cycloud-hosted runner 44 myshoes-proxy (Global IP) github.com GHES User Tenant A Tenant B webhook myshoes REST API Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service myshoes-proxy • HTTP Proxy • webhook from GitHub • myshoes REST API • Use svc.cluster.local • Prometheus metrics • Usage API for billing

Slide 45

Slide 45 text

Kubernetes Cluster Architecture of Cycloud-hosted runner 45 myshoes-proxy (Global IP) github.com GHES User Tenant A Tenant B webhook myshoes REST API Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service • shoes-lxd • shoes-provider for LXD • btrfs (CoW + dockerd) • btrfs subvolume • Mount NVMe volume

Slide 46

Slide 46 text

And more … 46

Slide 47

Slide 47 text

Kubernetes Cluster Architecture of Cycloud-hosted runner 47 myshoes-proxy (Global IP) github.com GHES User Tenant A Tenant B webhook myshoes REST API Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service

Slide 48

Slide 48 text

Architecture of Cycloud-hosted runner 48 Tenant A Tenant B Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service

Slide 49

Slide 49 text

Cycloud Architecture of Cycloud-hosted runner 49 Tenant A Tenant B Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service Service VM Service VM Service VM

Slide 50

Slide 50 text

Cycloud Architecture of Cycloud-hosted runner 50 Tenant A Tenant B Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service Service VM Service VM Service VM

Slide 51

Slide 51 text

Cycloud Architecture of Cycloud-hosted runner 51 Tenant A Tenant B Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) Stadium VM Stadium VM Stadium VMs LXD (listen 8 44 3 ) k 8 s Service k 8 s Service for Service X for Service Y k 8 s Service Tenant C k 8 s Service Service VM Service VM Service VM Cycloud user is …  Connected to my tenant is OK Anything complete in Cycloud!

Slide 52

Slide 52 text

Spec of Cycloud-hosted runner • Stadium as a Cycloud Virtual Machine • We plan to provide cost less than GitHub-hosted • runner image is almost the same as GitHub-hosted • https://github.com/whywaita/virtual-environments-lxd • + con fi gure apt repository, etc … • Spec of runner: 2 Core/ 20 GB ~ 1 6 Core/ 160 GB 52

Slide 53

Slide 53 text

Feature works • For OSS • more shoes provider, more Documents • Support `--ephemeral` • For Cycloud-hosted runner • GPU runner ? • macOS runner? 53

Slide 54

Slide 54 text

Conclusion • Explain about runner for GitHub Actions • Pros / Cons • whywaita/myshoes is Good solution ☺ • It can manage self-hosted runner usefully! • We will develop myshoes and   provide Cycloud-hosted runner • Please user report, Issue, Pull-Request ! 54

Slide 55

Slide 55 text

Appendix 1 : myshoes vs • actions-runner-controller/actions-runner-controller (top of star in awesome- runners) • runner as a Kubernetes Pod • philips-labs/terraform-aws-github-runner, github-developer/self-hosted- runners-anthos • runner as a Public cloud instances or managed service • whywaita/myshoes • runner as a selective (able to implement shoes-providers) • more information: https://jonico.github.io/awesome-runners/ 55

Slide 56

Slide 56 text

Appendix 2 : customize myshoes • myshoes has a few components • Implemented `interface()` in Go • Modify cmd.go (as main.go),   Get customized myshoes ! 56

Slide 57

Slide 57 text

Appendix 2 : customize myshoes 57 https://github.com/whywaita/myshoes/blob/ 5 d 2 2 108 c 2 d 7 d 50 b 83 f 9 7 993 a 05 69 b 8 860 48 d 1 c 7 9 /cmd/server/cmd.go

Slide 58

Slide 58 text

Appendix 2 : customize myshoes • myshoes components • datastore: Persistent datastore (MySQL) • safety: Check can creating (Unlimited) • runner manager: Delete old runners 58

Slide 59

Slide 59 text

Appendix 2 : customize myshoes 59 https://github.com/whywaita/myshoes/blob/ 5 d 221 08 c 2 d 7 d 50 b 83 f 97993 a 0 569 b 88 60 48 d 1 c 79 /pkg/starter/safety/safety.go

Slide 60

Slide 60 text

Appendix 2 : customize myshoes 60 https://github.com/whywaita/myshoes/blob/ 5 d 221 08 c 2 d 7 d 50 b 83 f 97993 a 0 569 b 88 60 48 d 1 c 79 /pkg/starter/starter.go

Slide 61

Slide 61 text

Appendix 2 : customize myshoes 61 https://github.com/whywaita/myshoes/blob/ 5 d 221 08 c 2 d 7 d 50 b 83 f 97993 a 0 569 b 88 60 48 d 1 c 79 /pkg/starter/safety/unlimited/unlimited.go

Slide 62

Slide 62 text

Appendix 2 : customize myshoes • Switch other Safety() implements • Reason of waiting … • For billing • For stadium resource • For Legal (like HTTP 45 1 ) 62