Development myshoes and Provide Cycloud-hosted runner -- GitHub Actions with your shoes. #cndjp

Talked by https://cnd.connpass.com/event/223825/

Original: https://www.slideshare.net/whywaita/development-myshoes-and-provide-cycloudhosted-runner-github-actions-with-your-shoes

This slide is re-upload by the author.

Transcript

1. GitHub Actions with your shoes.
   
   
   Development myshoes and
   

   Provide Cycloud-hosted runner
   CyberAgent, Inc.
   

   CyberAgent group Infrastructure Unit Private Cloud Team
   

   Cloud Maker / Nakanishi Kento @whywaita
   
   
   Cloud Native Developers #
   17
   僕たち私たちの CI/CD はこれだ！ @ ONLINE
   

   2021
   /
   1
   0
   /
   0
   8
   1
   

   View Slide

2. whywaita
   • Nakanishi Kento
   
   
   • CloudMaker in CyberAgent since
   2
   0 19 

   CIU Private Cloud team
   
   
   • Work: Dev and Ops using OpenStack / Kubernetes
   
   
   • Hobby: homelab and Anime-song
   
   
   • Become Founder of AS
   63 7
   9 0
   (CA-LAB) 🥰
   2
   

   View Slide

3. 1
   .Runner in GitHub Actions
   
   
   2
   .What’s myshoes
   
   
   3
   .myshoes in CyberAgent
   3
   

   View Slide

4. Runner in GitHub Actions
   4
   

   View Slide

5. GitHub Actions runner
   • GitHub-hosted runner
   
   
   • Managed by GitHub
   
   
   • Backend as a Azure Pipelines
   
   
   • self-hosted runner
   
   
   • Managed by you
   
   
   • github.com/actions/runner install to own server
   
   
   • On-prem, Cloud, homelab
   …
   5
   

   View Slide

6. GitHub-hosted runner
   • Pros
   
   
   • Fully managed
   
   
   • Cons
   
   
   • Fixed spec (CPU:
   2
   , Mem:
   7
   GB)
   
   
   • Need many money 💸
   6
   

   View Slide

7. self-hosted runner
   • Pros
   
   
   • Running own cloud, infrastructure
   
   
   • Using small cost and own IP address
   
   
   • Cons
   
   
   • Not Consistent (actions/runner is daemon)
   
   
   • “running terraform that managed self-hosted runner in self-
   hosted runner
   …
   ”
   7
   

   View Slide

8. 😩😩😩
   8
   

   View Slide

9. Our solution is
   …
   9
   

   View Slide

10. View Slide

11. 11
    What’s
    

    View Slide

12. whywaita/myshoes
    • Auto scaling self-hosted runner for GitHub Actions
    
    
    • Creating self-hosted runner per job
    
    
    • Available your infrastructure
    
    
    • shoes-aws, shoes-lxd, and you can implement
    
    
    • https://github.com/whywaita/myshoes
    12
    

    View Slide

13. Usage - Execute job
    13
    Execute job
    …
    

    View Slide

14. Usage - Creating runner
    14
    Auto add by myshoes
    

    View Slide

15. Usage - End job
    15
    End job
    …
    

    View Slide

16. Usage - Deleting runner
    16
    Auto delete by myshoes
    

    View Slide

17. Can use as a GitHub-hosted 😍
    17
    

    View Slide

18. Creating per job, Can use consistent
    18
    

    View Slide

19. Creating per job, Can use consistent
    19
    Creating a new runner per GitHub Actions work
    fl
    ows
    

    View Slide

20. Creating per job, Can use consistent
    Runner
    Creating a new runner per GitHub Actions work
    fl
    ows
    

    View Slide

21. Creating per job, Can use consistent
    Runner
    Runner
    Creating a new runner per GitHub Actions work
    fl
    ows
    

    View Slide

22. Creating per job, Can use consistent
    Runner
    Runner
    Runner
    Runner
    Runner
    Runner
    Creating a new runner per GitHub Actions work
    fl
    ows
    

    View Slide

23. Support matrix
    23
    

    View Slide

24. Support matrix
    24
    

    View Slide

25. Support matrix
    25
    

    View Slide

26. Support matrix
    26
    Runner
    Runner
    Creating a new runner per GitHub Actions jobs
    

    View Slide

27. Internal - Add runner
    27
    Cloud Provider
    
    
    • AWS / GCP / Azure
    
    
    • LXD
    
    
    • OpenStack
    
    
    • etc etc
    …
    ② Send a webhook
    

      to myshoes
    ① Request job
    ③ Execute provider
    Virtual Machine
    
    
    with actions/runner
    ④ Create an instance
    ⑤ Register runner
    

    View Slide

28. Internal - Delete runner
    28
    Cloud Provider
    
    
    • AWS / GCP / Azure
    
    
    • LXD
    
    
    • OpenStack
    
    
    • etc etc
    …
    ① Get a list of runner
    ③ Execute provider
    Virtual Machine
    
    
    with actions/runner
    ④ Delete an instance
    ② Check to delete runner
    

    View Slide

29. Internal - Add runner
    29
    Cloud Provider
    
    
    • AWS / GCP / Azure
    
    
    • LXD
    
    
    • OpenStack
    
    
    • etc etc
    …
    ② Send a webhook
    

      to myshoes
    ① Request job
    ③ Execute provider
    Virtual Machine
    
    
    with actions/runner
    ④ Create an instance
    ⑤ Register runner
    

    View Slide

30. Internal - Add runner
    30
    Cloud Provider
    
    
    • AWS / GCP / Azure
    
    
    • LXD
    
    
    • OpenStack
    
    
    • etc etc
    …
    ③ Execute provider
    Virtual Machine
    
    
    with actions/runner
    ④ Create an instance
    ② Send a webhook
    

    

    View Slide

31. Internal - shoes-provider
    31
    Cloud Provider
    Execute provider
    
    
    (hashicorp/go-plugin)
    Virtual Machine
    
    
    with actions/runner
    shoes-xxx
    Virtual Machine
    
    
    with actions/runner
    Virtual Machine
    
    
    with actions/runner
    Call API
    1
    .Runner Name : Name of runner
    
    
    2
    .Setup Script : Script of provisioning actions/runner
    
    
    3
    .Resource Type : Spec for runner
    

    View Slide

32. shoes-provider example: shoes-aws
    32
    Amazon Web Services
    Execute provider
    
    
    (hashicorp/go-plugin)
    EC
    2
    
    with actions/runner
    shoes-aws
    EC
    2
    
    with actions/runner
    EC
    2
    
    with actions/runner
    Call AWS REST API
    1
    .Runner Name to tag “Name”
    
    
    2
    .Setup Script to cloud-init user-data
    
    
    3
    .Resource Type to AWS Instance Type
    

    View Slide

33. shoes-provider example: shoes-lxd
    33
    LXD server
    Execute provider
    
    
    (hashicorp/go-plugin)
    LXD Container
    
    
    with actions/runner
    shoes-lxd
    Call LXD REST API
    LXD Container
    
    
    with actions/runner
    LXD Container
    
    
    with actions/runner
    1
    .Runner Name to Instance name
    
    
    2
    .Setup Script to cloud-init user-data
    
    
    3
    .Resource Type to limits.cpu, limits.memory
    

    View Slide

34. myshoes-provider has dream 🤩
    • Cheap instance in Public cloud
    
    
    • AWS Spot Instances, GCP Preemptible Instances
    
    
    • Private Cloud, homelab, On-prem
    
    
    • LXD, OpenStack
    
    
    • Speci
    fi
    c hardwares
    
    
    • GPU, FPGA
    34
    

    View Slide

35. myshoes in CyberAgent
    35
    

    View Slide

36. Cycloud
    • Cy(berAgent)cloud
    
    
    • Private Cloud
    
    
    • IaaS, Managed Service
    
    
    • Customized for CyberAgent products
    
    
    • CA products can use Compute at a low cost 💰
    36
    

    View Slide

37. CI in CyberAgent
    • Products decide to use technology, CI/CD
    
    
    • github.com , GitHub Enterprise Server, GitLab
    
    
    • Jenkins, Travis CI, CircleCI, Drone CI, Bitrise
    
    
    • In many case, use enterprise application
    

    ex:) CircleCI Enterprise
    
    
    • A few products start to use GitHub Actions in beta
    37
    

    View Slide

38. CI in CyberAgent
    • Need to migrate Circle CI Enterprise to GitHub Actions
    
    
    • End of using Enterprise
    
    
    • Bye cost merit
    
    
    • Increase support cost
    
    
    • whywaita “We can get a cost down if use Cycloud?”
    
    
    → Start to develop myshoes
    …
    38
    

    View Slide

39. myshoes in CyberAgent
    39
    

    View Slide

40. myshoes in CyberAgent (
    20
    2
    1
    /
    0
    9
    ~)
    40
    

    View Slide

41. myshoes in CyberAgent
    • Cycloud-hosted runner has released 🥳
    
    
    • Managed service myshoes
    
    
    • Support GHES, GHEC, github.com
    
    
    • Cover many use cases in CyberAgent
    
    
    • Multi-tenancy
    41
    

    View Slide

42. myshoes in CyberAgent
    • whywaita/myshoes + shoes-lxd
    
    
    • multi-tenancy: myshoes-controller
    
    
    • Implemented Kubernetes Controller
    
    
    • MySQL Cluster (with orchestrator)
    
    
    • Docker Registry v
    2
    , LXD image server
    42
    

    View Slide

43. Kubernetes Cluster
    Architecture of Cycloud-hosted runner
    43
    myshoes-proxy
    
    
    (Global IP)
    github.com
    
    
    GHES
    User
    Tenant A
    Tenant B
    webhook
    myshoes REST API
    Stadium VM
    Stadium VM
    Stadium VMs
    
    
    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs
    
    
    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs
    
    
    LXD (listen
    8
    44 3
    )
    k
    8
    s Service
    k
    8
    s Service
    for Service X
    for Service Y
    k
    8
    s Service
    Tenant C
    k
    8
    s Service
    

    View Slide

44. Kubernetes Cluster
    Architecture of Cycloud-hosted runner
    44
    myshoes-proxy
    
    
    (Global IP)
    github.com
    
    
    GHES
    User
    Tenant A
    Tenant B
    webhook
    myshoes REST API
    Stadium VM
    Stadium VM
    Stadium VMs
    
    
    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs
    
    
    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs
    
    
    LXD (listen
    8
    44 3
    )
    k
    8
    s Service
    k
    8
    s Service
    for Service X
    for Service Y
    k
    8
    s Service
    Tenant C
    k
    8
    s Service
    myshoes-proxy
    
    
    • HTTP Proxy
    
    
    • webhook from GitHub
    
    
    • myshoes REST API
    
    
    • Use svc.cluster.local
    
    
    • Prometheus metrics
    
    
    • Usage API for billing
    

    View Slide

45. Kubernetes Cluster
    Architecture of Cycloud-hosted runner
    45
    myshoes-proxy
    
    
    (Global IP)
    github.com
    
    
    GHES
    User
    Tenant A
    Tenant B
    webhook
    myshoes REST API
    Stadium VM
    Stadium VM
    Stadium VMs
    
    
    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs
    
    
    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs
    
    
    LXD (listen
    8
    44 3
    )
    k
    8
    s Service
    k
    8
    s Service
    for Service X
    for Service Y
    k
    8
    s Service
    Tenant C
    k
    8
    s Service
    • shoes-lxd
    
    
    • shoes-provider for LXD
    
    
    • btrfs (CoW + dockerd)
    
    
    • btrfs subvolume
    
    
    • Mount NVMe volume
    

    View Slide

46. And more
    …
    46
    

    View Slide

47. Kubernetes Cluster
    Architecture of Cycloud-hosted runner
    47
    myshoes-proxy
    
    
    (Global IP)
    github.com
    
    
    GHES
    User
    Tenant A
    Tenant B
    webhook
    myshoes REST API
    Stadium VM
    Stadium VM
    Stadium VMs
    
    
    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs
    
    
    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs
    
    
    LXD (listen
    8
    44 3
    )
    k
    8
    s Service
    k
    8
    s Service
    for Service X
    for Service Y
    k
    8
    s Service
    Tenant C
    k
    8
    s Service
    

    View Slide

48. Architecture of Cycloud-hosted runner
    48
    Tenant A
    Tenant B
    Stadium VM
    Stadium VM
    Stadium VMs
    
    
    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs
    
    
    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs
    
    
    LXD (listen
    8
    44 3
    )
    k
    8
    s Service
    k
    8
    s Service
    for Service X
    for Service Y
    k
    8
    s Service
    Tenant C
    k
    8
    s Service
    

    View Slide

49. Cycloud
    Architecture of Cycloud-hosted runner
    49
    Tenant A
    Tenant B
    Stadium VM
    Stadium VM
    Stadium VMs
    
    
    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs
    
    
    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs
    
    
    LXD (listen
    8
    44 3
    )
    k
    8
    s Service
    k
    8
    s Service
    for Service X
    for Service Y
    k
    8
    s Service
    Tenant C
    k
    8
    s Service
    Service VM
    Service VM
    Service VM
    

    View Slide

50. Cycloud
    Architecture of Cycloud-hosted runner
    50
    Tenant A
    Tenant B
    Stadium VM
    Stadium VM
    Stadium VMs
    
    
    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs
    
    
    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs
    
    
    LXD (listen
    8
    44 3
    )
    k
    8
    s Service
    k
    8
    s Service
    for Service X
    for Service Y
    k
    8
    s Service
    Tenant C
    k
    8
    s Service
    Service VM
    Service VM
    Service VM
    

    View Slide

51. Cycloud
    Architecture of Cycloud-hosted runner
    51
    Tenant A
    Tenant B
    Stadium VM
    Stadium VM
    Stadium VMs
    
    
    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs
    
    
    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs
    
    
    LXD (listen
    8
    44 3
    )
    k
    8
    s Service
    k
    8
    s Service
    for Service X
    for Service Y
    k
    8
    s Service
    Tenant C
    k
    8
    s Service
    Service VM
    Service VM
    Service VM
    Cycloud user is
    …

    Connected to my tenant is OK
    
    
    Anything complete in Cycloud!
    

    View Slide

52. Spec of Cycloud-hosted runner
    • Stadium as a Cycloud Virtual Machine
    
    
    • We plan to provide cost less than GitHub-hosted
    
    
    • runner image is almost the same as GitHub-hosted
    
    
    • https://github.com/whywaita/virtual-environments-lxd
    
    
    • + con
    fi
    gure apt repository, etc
    …
    
    • Spec of runner:
    2
    Core/
    20
    GB ~
    1 6
    Core/
    160
    GB
    52
    

    View Slide

53. Feature works
    • For OSS
    
    
    • more shoes provider, more Documents
    
    
    • Support `--ephemeral`
    
    
    • For Cycloud-hosted runner
    
    
    • GPU runner ?
    
    
    • macOS runner?
    53
    

    View Slide

54. Conclusion
    • Explain about runner for GitHub Actions
    
    
    • Pros / Cons
    
    
    • whywaita/myshoes is Good solution ☺
    
    
    • It can manage self-hosted runner usefully!
    
    
    • We will develop myshoes and
    

    provide Cycloud-hosted runner
    
    
    • Please user report, Issue, Pull-Request !
    54
    

    View Slide

55. Appendix
    1
    : myshoes vs
    • actions-runner-controller/actions-runner-controller (top of star in awesome-
    runners)
    
    
    • runner as a Kubernetes Pod
    
    
    • philips-labs/terraform-aws-github-runner, github-developer/self-hosted-
    runners-anthos
    
    
    • runner as a Public cloud instances or managed service
    
    
    • whywaita/myshoes
    
    
    • runner as a selective (able to implement shoes-providers)
    
    
    • more information: https://jonico.github.io/awesome-runners/
    55
    

    View Slide

56. Appendix
    2
    : customize myshoes
    • myshoes has a few components
    
    
    • Implemented `interface()` in Go
    
    
    • Modify cmd.go (as main.go),
    

    Get customized myshoes !
    56
    

    View Slide

57. Appendix
    2
    : customize myshoes
    57
    https://github.com/whywaita/myshoes/blob/
    5
    d
    2
    2 108
    c
    2
    d
    7
    d
    50
    b
    83
    f
    9
    7 993
    a
    05
    69
    b
    8
    860
    48
    d
    1
    c
    7
    9
    /cmd/server/cmd.go
    

    View Slide

58. Appendix
    2
    : customize myshoes
    • myshoes components
    
    
    • datastore: Persistent datastore (MySQL)
    
    
    • safety: Check can creating (Unlimited)
    
    
    • runner manager: Delete old runners
    58
    

    View Slide

59. Appendix
    2
    : customize myshoes
    59
    https://github.com/whywaita/myshoes/blob/
    5
    d
    221
    08
    c
    2
    d
    7
    d
    50
    b
    83
    f
    97993
    a
    0
    569
    b
    88
    60 48
    d
    1
    c
    79
    /pkg/starter/safety/safety.go
    

    View Slide

60. Appendix
    2
    : customize myshoes
    60
    https://github.com/whywaita/myshoes/blob/
    5
    d
    221
    08
    c
    2
    d
    7
    d
    50
    b
    83
    f
    97993
    a
    0
    569
    b
    88
    60 48
    d
    1
    c
    79
    /pkg/starter/starter.go
    

    View Slide

61. Appendix
    2
    : customize myshoes
    61
    https://github.com/whywaita/myshoes/blob/
    5
    d
    221
    08
    c
    2
    d
    7
    d
    50
    b
    83
    f
    97993
    a
    0
    569
    b
    88
    60 48
    d
    1
    c
    79
    /pkg/starter/safety/unlimited/unlimited.go
    

    View Slide

62. Appendix
    2
    : customize myshoes
    • Switch other Safety() implements
    
    
    • Reason of waiting
    …
    
    • For billing
    
    
    • For stadium resource
    
    
    • For Legal (like HTTP
    45 1
    )
    62
    

    View Slide