$30 off During Our Annual Pro Sale. View Details »

Development myshoes and Provide Cycloud-hosted runner -- GitHub Actions with your shoes. #cndjp

Development myshoes and Provide Cycloud-hosted runner -- GitHub Actions with your shoes. #cndjp

whywaita
PRO

October 12, 2021
Tweet

More Decks by whywaita

Other Decks in Technology

Transcript

  1. GitHub Actions with your shoes.


    Development myshoes and

    Provide Cycloud-hosted runner
    CyberAgent, Inc.

    CyberAgent group Infrastructure Unit Private Cloud Team

    Cloud Maker / Nakanishi Kento @whywaita


    Cloud Native Developers #
    17
    僕たち私たちの CI/CD はこれだ! @ ONLINE

    2021
    /
    1
    0
    /
    0
    8
    1

    View Slide

  2. whywaita
    • Nakanishi Kento


    • CloudMaker in CyberAgent since
    2
    0 19 

    CIU Private Cloud team


    • Work: Dev and Ops using OpenStack / Kubernetes


    • Hobby: homelab and Anime-song


    • Become Founder of AS
    63 7
    9 0
    (CA-LAB) 🥰
    2

    View Slide

  3. 1
    .Runner in GitHub Actions


    2
    .What’s myshoes


    3
    .myshoes in CyberAgent
    3

    View Slide

  4. Runner in GitHub Actions
    4

    View Slide

  5. GitHub Actions runner
    • GitHub-hosted runner


    • Managed by GitHub


    • Backend as a Azure Pipelines


    • self-hosted runner


    • Managed by you


    • github.com/actions/runner install to own server


    • On-prem, Cloud, homelab

    5

    View Slide

  6. GitHub-hosted runner
    • Pros


    • Fully managed


    • Cons


    • Fixed spec (CPU:
    2
    , Mem:
    7
    GB)


    • Need many money 💸
    6

    View Slide

  7. self-hosted runner
    • Pros


    • Running own cloud, infrastructure


    • Using small cost and own IP address


    • Cons


    • Not Consistent (actions/runner is daemon)


    • “running terraform that managed self-hosted runner in self-
    hosted runner


    7

    View Slide

  8. 😩😩😩
    8

    View Slide

  9. Our solution is

    9

    View Slide

  10. 10

    View Slide

  11. 11
    What’s

    View Slide

  12. whywaita/myshoes
    • Auto scaling self-hosted runner for GitHub Actions


    • Creating self-hosted runner per job


    • Available your infrastructure


    • shoes-aws, shoes-lxd, and you can implement


    • https://github.com/whywaita/myshoes
    12

    View Slide

  13. Usage - Execute job
    13
    Execute job

    View Slide

  14. Usage - Creating runner
    14
    Auto add by myshoes

    View Slide

  15. Usage - End job
    15
    End job

    View Slide

  16. Usage - Deleting runner
    16
    Auto delete by myshoes

    View Slide

  17. Can use as a GitHub-hosted 😍
    17

    View Slide

  18. Creating per job, Can use consistent
    18

    View Slide

  19. Creating per job, Can use consistent
    19
    Creating a new runner per GitHub Actions work
    fl
    ows

    View Slide

  20. Creating per job, Can use consistent
    Runner
    Creating a new runner per GitHub Actions work
    fl
    ows

    View Slide

  21. Creating per job, Can use consistent
    Runner
    Runner
    Creating a new runner per GitHub Actions work
    fl
    ows

    View Slide

  22. Creating per job, Can use consistent
    Runner
    Runner
    Runner
    Runner
    Runner
    Runner
    Creating a new runner per GitHub Actions work
    fl
    ows

    View Slide

  23. Support matrix
    23

    View Slide

  24. Support matrix
    24

    View Slide

  25. Support matrix
    25

    View Slide

  26. Support matrix
    26
    Runner
    Runner
    Creating a new runner per GitHub Actions jobs

    View Slide

  27. Internal - Add runner
    27
    Cloud Provider


    • AWS / GCP / Azure


    • LXD


    • OpenStack


    • etc etc

    ② Send a webhook

      to myshoes
    ① Request job
    ③ Execute provider
    Virtual Machine


    with actions/runner
    ④ Create an instance
    ⑤ Register runner

    View Slide

  28. Internal - Delete runner
    28
    Cloud Provider


    • AWS / GCP / Azure


    • LXD


    • OpenStack


    • etc etc

    ① Get a list of runner
    ③ Execute provider
    Virtual Machine


    with actions/runner
    ④ Delete an instance
    ② Check to delete runner

    View Slide

  29. Internal - Add runner
    29
    Cloud Provider


    • AWS / GCP / Azure


    • LXD


    • OpenStack


    • etc etc

    ② Send a webhook

      to myshoes
    ① Request job
    ③ Execute provider
    Virtual Machine


    with actions/runner
    ④ Create an instance
    ⑤ Register runner

    View Slide

  30. Internal - Add runner
    30
    Cloud Provider


    • AWS / GCP / Azure


    • LXD


    • OpenStack


    • etc etc

    ③ Execute provider
    Virtual Machine


    with actions/runner
    ④ Create an instance
    ② Send a webhook

    View Slide

  31. Internal - shoes-provider
    31
    Cloud Provider
    Execute provider


    (hashicorp/go-plugin)
    Virtual Machine


    with actions/runner
    shoes-xxx
    Virtual Machine


    with actions/runner
    Virtual Machine


    with actions/runner
    Call API
    1
    .Runner Name : Name of runner


    2
    .Setup Script : Script of provisioning actions/runner


    3
    .Resource Type : Spec for runner

    View Slide

  32. shoes-provider example: shoes-aws
    32
    Amazon Web Services
    Execute provider


    (hashicorp/go-plugin)
    EC
    2

    with actions/runner
    shoes-aws
    EC
    2

    with actions/runner
    EC
    2

    with actions/runner
    Call AWS REST API
    1
    .Runner Name to tag “Name”


    2
    .Setup Script to cloud-init user-data


    3
    .Resource Type to AWS Instance Type

    View Slide

  33. shoes-provider example: shoes-lxd
    33
    LXD server
    Execute provider


    (hashicorp/go-plugin)
    LXD Container


    with actions/runner
    shoes-lxd
    Call LXD REST API
    LXD Container


    with actions/runner
    LXD Container


    with actions/runner
    1
    .Runner Name to Instance name


    2
    .Setup Script to cloud-init user-data


    3
    .Resource Type to limits.cpu, limits.memory

    View Slide

  34. myshoes-provider has dream 🤩
    • Cheap instance in Public cloud


    • AWS Spot Instances, GCP Preemptible Instances


    • Private Cloud, homelab, On-prem


    • LXD, OpenStack


    • Speci
    fi
    c hardwares


    • GPU, FPGA
    34

    View Slide

  35. myshoes in CyberAgent
    35

    View Slide

  36. Cycloud
    • Cy(berAgent)cloud


    • Private Cloud


    • IaaS, Managed Service


    • Customized for CyberAgent products


    • CA products can use Compute at a low cost 💰
    36

    View Slide

  37. CI in CyberAgent
    • Products decide to use technology, CI/CD


    • github.com , GitHub Enterprise Server, GitLab


    • Jenkins, Travis CI, CircleCI, Drone CI, Bitrise


    • In many case, use enterprise application

    ex:) CircleCI Enterprise


    • A few products start to use GitHub Actions in beta
    37

    View Slide

  38. CI in CyberAgent
    • Need to migrate Circle CI Enterprise to GitHub Actions


    • End of using Enterprise


    • Bye cost merit


    • Increase support cost


    • whywaita “We can get a cost down if use Cycloud?”


    → Start to develop myshoes

    38

    View Slide

  39. myshoes in CyberAgent
    39

    View Slide

  40. myshoes in CyberAgent (
    20
    2
    1
    /
    0
    9
    ~)
    40

    View Slide

  41. myshoes in CyberAgent
    • Cycloud-hosted runner has released 🥳


    • Managed service myshoes


    • Support GHES, GHEC, github.com


    • Cover many use cases in CyberAgent


    • Multi-tenancy
    41

    View Slide

  42. myshoes in CyberAgent
    • whywaita/myshoes + shoes-lxd


    • multi-tenancy: myshoes-controller


    • Implemented Kubernetes Controller


    • MySQL Cluster (with orchestrator)


    • Docker Registry v
    2
    , LXD image server
    42

    View Slide

  43. Kubernetes Cluster
    Architecture of Cycloud-hosted runner
    43
    myshoes-proxy


    (Global IP)
    github.com


    GHES
    User
    Tenant A
    Tenant B
    webhook
    myshoes REST API
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    k
    8
    s Service
    k
    8
    s Service
    for Service X
    for Service Y
    k
    8
    s Service
    Tenant C
    k
    8
    s Service

    View Slide

  44. Kubernetes Cluster
    Architecture of Cycloud-hosted runner
    44
    myshoes-proxy


    (Global IP)
    github.com


    GHES
    User
    Tenant A
    Tenant B
    webhook
    myshoes REST API
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    k
    8
    s Service
    k
    8
    s Service
    for Service X
    for Service Y
    k
    8
    s Service
    Tenant C
    k
    8
    s Service
    myshoes-proxy


    • HTTP Proxy


    • webhook from GitHub


    • myshoes REST API


    • Use svc.cluster.local


    • Prometheus metrics


    • Usage API for billing

    View Slide

  45. Kubernetes Cluster
    Architecture of Cycloud-hosted runner
    45
    myshoes-proxy


    (Global IP)
    github.com


    GHES
    User
    Tenant A
    Tenant B
    webhook
    myshoes REST API
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    k
    8
    s Service
    k
    8
    s Service
    for Service X
    for Service Y
    k
    8
    s Service
    Tenant C
    k
    8
    s Service
    • shoes-lxd


    • shoes-provider for LXD


    • btrfs (CoW + dockerd)


    • btrfs subvolume


    • Mount NVMe volume

    View Slide

  46. And more

    46

    View Slide

  47. Kubernetes Cluster
    Architecture of Cycloud-hosted runner
    47
    myshoes-proxy


    (Global IP)
    github.com


    GHES
    User
    Tenant A
    Tenant B
    webhook
    myshoes REST API
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    k
    8
    s Service
    k
    8
    s Service
    for Service X
    for Service Y
    k
    8
    s Service
    Tenant C
    k
    8
    s Service

    View Slide

  48. Architecture of Cycloud-hosted runner
    48
    Tenant A
    Tenant B
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    k
    8
    s Service
    k
    8
    s Service
    for Service X
    for Service Y
    k
    8
    s Service
    Tenant C
    k
    8
    s Service

    View Slide

  49. Cycloud
    Architecture of Cycloud-hosted runner
    49
    Tenant A
    Tenant B
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    k
    8
    s Service
    k
    8
    s Service
    for Service X
    for Service Y
    k
    8
    s Service
    Tenant C
    k
    8
    s Service
    Service VM
    Service VM
    Service VM

    View Slide

  50. Cycloud
    Architecture of Cycloud-hosted runner
    50
    Tenant A
    Tenant B
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    k
    8
    s Service
    k
    8
    s Service
    for Service X
    for Service Y
    k
    8
    s Service
    Tenant C
    k
    8
    s Service
    Service VM
    Service VM
    Service VM

    View Slide

  51. Cycloud
    Architecture of Cycloud-hosted runner
    51
    Tenant A
    Tenant B
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    k
    8
    s Service
    k
    8
    s Service
    for Service X
    for Service Y
    k
    8
    s Service
    Tenant C
    k
    8
    s Service
    Service VM
    Service VM
    Service VM
    Cycloud user is
    …

    Connected to my tenant is OK


    Anything complete in Cycloud!

    View Slide

  52. Spec of Cycloud-hosted runner
    • Stadium as a Cycloud Virtual Machine


    • We plan to provide cost less than GitHub-hosted


    • runner image is almost the same as GitHub-hosted


    • https://github.com/whywaita/virtual-environments-lxd


    • + con
    fi
    gure apt repository, etc


    • Spec of runner:
    2
    Core/
    20
    GB ~
    1 6
    Core/
    160
    GB
    52

    View Slide

  53. Feature works
    • For OSS


    • more shoes provider, more Documents


    • Support `--ephemeral`


    • For Cycloud-hosted runner


    • GPU runner ?


    • macOS runner?
    53

    View Slide

  54. Conclusion
    • Explain about runner for GitHub Actions


    • Pros / Cons


    • whywaita/myshoes is Good solution ☺


    • It can manage self-hosted runner usefully!


    • We will develop myshoes and

    provide Cycloud-hosted runner


    • Please user report, Issue, Pull-Request !
    54

    View Slide

  55. Appendix
    1
    : myshoes vs
    • actions-runner-controller/actions-runner-controller (top of star in awesome-
    runners)


    • runner as a Kubernetes Pod


    • philips-labs/terraform-aws-github-runner, github-developer/self-hosted-
    runners-anthos


    • runner as a Public cloud instances or managed service


    • whywaita/myshoes


    • runner as a selective (able to implement shoes-providers)


    • more information: https://jonico.github.io/awesome-runners/
    55

    View Slide

  56. Appendix
    2
    : customize myshoes
    • myshoes has a few components


    • Implemented `interface()` in Go


    • Modify cmd.go (as main.go),

    Get customized myshoes !
    56

    View Slide

  57. Appendix
    2
    : customize myshoes
    57
    https://github.com/whywaita/myshoes/blob/
    5
    d
    2
    2 108
    c
    2
    d
    7
    d
    50
    b
    83
    f
    9
    7 993
    a
    05
    69
    b
    8
    860
    48
    d
    1
    c
    7
    9
    /cmd/server/cmd.go

    View Slide

  58. Appendix
    2
    : customize myshoes
    • myshoes components


    • datastore: Persistent datastore (MySQL)


    • safety: Check can creating (Unlimited)


    • runner manager: Delete old runners
    58

    View Slide

  59. Appendix
    2
    : customize myshoes
    59
    https://github.com/whywaita/myshoes/blob/
    5
    d
    221
    08
    c
    2
    d
    7
    d
    50
    b
    83
    f
    97993
    a
    0
    569
    b
    88
    60 48
    d
    1
    c
    79
    /pkg/starter/safety/safety.go

    View Slide

  60. Appendix
    2
    : customize myshoes
    60
    https://github.com/whywaita/myshoes/blob/
    5
    d
    221
    08
    c
    2
    d
    7
    d
    50
    b
    83
    f
    97993
    a
    0
    569
    b
    88
    60 48
    d
    1
    c
    79
    /pkg/starter/starter.go

    View Slide

  61. Appendix
    2
    : customize myshoes
    61
    https://github.com/whywaita/myshoes/blob/
    5
    d
    221
    08
    c
    2
    d
    7
    d
    50
    b
    83
    f
    97993
    a
    0
    569
    b
    88
    60 48
    d
    1
    c
    79
    /pkg/starter/safety/unlimited/unlimited.go

    View Slide

  62. Appendix
    2
    : customize myshoes
    • Switch other Safety() implements


    • Reason of waiting


    • For billing


    • For stadium resource


    • For Legal (like HTTP
    45 1
    )
    62

    View Slide