Slide 1

Slide 1 text

1 Serving GitHub Actions Self-Hosted Runner as a Platform Merpay・Mercoin SRE @keke Merpay SRE Tech Talk 2021/11/05

Slide 2

Slide 2 text

2 Agenda 01 ● Introduction to GitHub Actions Self-Hosted Runner (Part 1) 02 ● Serve as a Platform (Part 2) Today is part 1 😄

Slide 3

Slide 3 text

3 01 Introduction to GitHub Actions Self-Hosted Runner (Part 1)

Slide 4

Slide 4 text

4 What’s GitHub Actions? GitHub native continuous Integration (CI) tool which is triggered on GitHub events (e.g., Pull request creation, comment on an issue, etc.) A “Workflow” has “Job(s)” that runs an “Action”, which is offered step officially by third-party or by yourself.

Slide 5

Slide 5 text

5 Workflow

Slide 6

Slide 6 text

6 How it works? Developers GitHub Runs Workflow on Runners ② Operation ① Listen to a queue ( HTTP long polling) ⑤ Results & Updates Commit Status ③ Dispatch to queue GitHub Queue ④ Get event

Slide 7

Slide 7 text

7 Types of Runners Cloud Hosted Self Hosted

Slide 8

Slide 8 text

8 Specify Runner types 👈

Slide 9

Slide 9 text

9 Types of Runners 1. Cloud Hosted (GitHub Managed) Runners Users doesn’t need to prepare anything (instead of money 💰) Developers GitHub Runs Workflow on Runners ② Operation ① Listen to a queue ( HTTP long polling) ⑤ Results & Updates Commit Status ③ Dispatch to queue GitHub Queue ④ Get event Managed by GitHub

Slide 10

Slide 10 text

10 ● Windows Server ○ 2022 ○ 2019 ● macOS ○ Big Sur 11 ○ Catalina ● Ubuntu ○ 20.04 ○ 18.04 Installed software are limited. OS Major Limitations of Cloud Hosted Runners IP addresses are not static in GitHub organization level, shared by other organizations. Can’t configure good firewall. GitHub can have IP restrictions on organization so the runners can’t access GitHub.
 Ref: Access GitHub Meta API for IP addresses (there are many 😄) 
 $ curl https://api.github.com/meta Network Egress Only offers one machine type ● 2 CPU core
 ● 7GB RAM
 ● 14GB SSD
 Sometimes there's too much or too little, not cost efficient. For example, big Go projects wants CPU specific machines while Next.js project wants memory specific machines.
 Hardware ● About GitHub-hosted runners, https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners ● GItHub Actions Virtual Environments, https://github.com/actions/virtual-environments

Slide 11

Slide 11 text

11 Types of Runners 1. Self-Hosted Runners (actions/runner) GitHub only dispatches event to the GitHub queue. Developers GitHub Runs Workflow on Runners ② Operation ① Register runner, Listen to a queue ⑤ Results & Updates Commit Status ③ Dispatch to queue GitHub Queue ④ Get event Managed by users

Slide 12

Slide 12 text

12 Organization Level 3 Types of Self-Hosted Runners Repository Level 
 Enterprise Level Repo A Repo B Repo Z Repo W Org A Org P Repo A Repo B Repo Z Repo W Org A Org P Repo A Repo B Repo C

Slide 13

Slide 13 text

13 Any 😄 OS Major advantages of Self-Hosted Runners Send egress thought NAT, VPN, etc. depends on user 😄 (Can make IP static) Network Egress Any 😄
 Hardware

Slide 14

Slide 14 text

14 Hard to manage efficiently. Scalability Major disadvantages of Self-Hosted Runners Two big problems: ● Need to register the Runner to organization, repository, etc ● Need to managed the Runner for each organizations, repository, etc Provisioning (Platform) Hard to provision the runners. Sometimes we want to offer many machine types: ● Standard ● CPU specific ● Memory specific ● “Big” machines like other cloud provider products (GCP Cloud Build, AWS EC2, etc) Provisioning (Machines) Actual metrics


Slide 15

Slide 15 text

15 How to manage it? Disclaimer: Following slides include things that have already been implemented or are planned to be implemented. As such, they may be changed later or not implemented yet.

Slide 16

Slide 16 text

16 Provisioning Runner’s Machines Offer repository level runners (currently organization level) to isolate the runners by microservices. Runs a dedicated GKE cluster with actions-runner-controller/actions-runner-controller, a Kubernetes controller to manage Self-Hosted Runners. The runners created by this controller’s CRD RunnerDeployment is backed by StatefulSets. On runner start, the runner will access the GitHub API and register the runner to the repository.

Slide 17

Slide 17 text

17 Kubernetes Manifest to provision runners

Slide 18

Slide 18 text

18 Provisioning Machines variations Cloud Hosted Runners only provides one machine type (2 CPU cores, 7GB RAM, 14GB SSD) while the Self-Hosted Runners can be any machine type by adding labels. 1 CPU core 100GB RAM standard memory 20 CPU core 20GB RAM high 0.1 CPU core 1GB RAM tiny 👉 Big Next.js App 👉 Docker builds 👉 Closing stale PRs

Slide 19

Slide 19 text

19 Runners with labels 👈

Slide 20

Slide 20 text

20 Specify Runner by labels Labels

Slide 21

Slide 21 text

21 Scaling Runners actions-runner-controller/actions-runner-controller also offers a webhook server and a CRD HorizontalRunnerAutoscaler (HRA) that receive the webhook event and scale the runners (similar to Kubernetes HorizontalPodAutoscaler(HPA)).

Slide 22

Slide 22 text

22 Scaling Runners Developers GitHub Runners ① Operation ② Webhook (workflow_job) ③ Create Runner Webhook Server ② Dispatch to queue GitHub Queue ④ Register, listen to queue ⑤ Event

Slide 23

Slide 23 text

23 Network Egress For security reasons, some GitHub Organizations has IP restriction on GitHub Organization. Cloud Hosted Runners has wide IP range and scope to the GitHub not the organization. Like normal “microservices” Egress of Self-Hosted Runners static public IP made possible by Cloud NAT + Cloud Router.

Slide 24

Slide 24 text

24 ✅ Provisioning Runners variations ✅ Scalability of Runners ✅ GitHub IP restrictions

Slide 25

Slide 25 text

25 02 Serve as a Platform (Part 2) Next Time 🌕 🚀 Some topics ・How to manage the tenants on the Kubernetes? ・How to automatically register the repository runners? ・Integration with Starter Kits?

Slide 26

Slide 26 text

26 Thank you!