$30 off During Our Annual Pro Sale. View Details »

GitHub Actions runner基盤におけるオンプレミスマルチテナントアプリケーションの運用 #CADC2022

GitHub Actions runner基盤におけるオンプレミスマルチテナントアプリケーションの運用 #CADC2022

CyberAgent Developer Conference 2022 の登壇資料です https://cadc.cyberagent.co.jp/2022/program/github-actions-runner-oss-myshoes/

CIU (CyberAgent Infrastructure Unit) ではCycloudとよばれるプライベートクラウドを運用しており、2021年9月にGitHub Actiionsにおけるrunner基盤である Cycloud-hosted runner を提供開始しました。これは安価にかつ強力なGitHub Actions runnerを社内に向けて提供するものです。

GitHub-hostedと同様に動的なrunner作成を行い、既存のプライベートクラウド環境とハイブリッドな運用を行いながらもマルチテナントを提供するためにさまざまなレイヤーから取り組みを行っています。

本登壇では動的なrunner作成を行うOSSであるmyshoesについて紹介した上で、myshoesを利用したマネージドサービスCycloud-hosted runnerの技術仕様に触れ、なぜその仕様が必要だったのか、その仕様を実現するためにどのように取り組んだかについてお話します。

whywaita
PRO

March 24, 2022
Tweet

More Decks by whywaita

Other Decks in Technology

Transcript

  1. (JU)VC"DUJPOTSVOOFSج൫ʹ͓͚Δ

    ΦϯϓϨϛε

    ϚϧνςφϯτΞϓϦέʔγϣϯͷӡ༻
    ݱঢ়ࡱӨޙͷεϥΠυࠩ͠ସ͑
    ˞Ծஔ͖

    View Slide

  2. த੢ݐొXIZXBJUB
    גࣜձࣾαΠόʔΤʔδΣϯτ೥৽ଔೖࣾ
    $ZCFS"HFOUHSPVQ*OGSBTUSVDUVSF6OJU $*6


    ɹ1MBUGPSNEJW։ൃνʔϜ
    w ۀ຿$"શࣾʹ޲͚ͨϓϥΠϕʔτΫϥ΢υ։ൃӡ༻
    w *BB4 4BB4CZ0QFO4UBDL ,VCFSOFUFT FUD
    w झຯࣗ୐αʔόɺຑ੃ ࢝Ίͨ
    ɺඒຯ͍͠LDVQ͕͞͠

    View Slide

  3. $POUFOUT
    ͸͡Ίʹ
    w $ZDMPVE
    w NZTIPFT $ZDMPVEIPTUFESVOOFS
    $ZDMPVEIPTUFESVOOFSʹ͓͚Δ

    Ϛϧνςφϯγʔ
    #VJME6Q
    ܧଓతͳ#VJME6Q

    View Slide

  4. ͸͡Ίʹ

    View Slide

  5. $ZDMPVE
    w$Z CFS"HFOU
    DMPVE
    wϓϥΠϕʔτΫϥ΢υ
    w *BB4,BB41BB44BB4
    w $"޲͚ΧελϚΠζࡁΈ
    w $16 .FNPSZ %JTL
    w҆ՁʹܭࢉػࢿݯΛར༻Մೳ

    View Slide

  6. XIZXBJUBNZTIPFT
    w"VUPTDBMJOH7JSUVBM.BDIJOFɹɹɹ
    SVOOFS🏃GPS(JU)VC"DUJPOT
    w(JU)VC"DUJPOTʹ͓͚ΔSVOOFSͷ

    ɹಈత࡞੒࡟আ
    wQSPWJEFSํࣜͰෳ਺ͷΫϥ΢υ্ʹ

    ɹSVOOFSΛσϓϩΠՄೳ

    View Slide

  7. Virtual Machine


    with actions/runner
    Cloud Provider


    • AWS / GCP / Azure


    • LXD / OpenStack


    • etc etc...
    ① Start job
    ② webhook
    ③ CreateInstance
    ④ Create
    ⑤ Register

    View Slide

  8. ৄࡉ͸ԼهൃදΛ֬͝ೝ͍ͩ͘͞
    IUUQTXXXTMJEFTIBSFOFUXIZXBJUBEFWFMPQNFOUNZTIPFTBOEQSPWJEF
    DZDMPVEIPTUFESVOOFSHJUIVCBDUJPOTXJUIZPVSTIPFT

    View Slide

  9. ͜ΕΛ࢖ͬͨαʔϏε͕

    View Slide

  10. View Slide

  11. $ZDMPVEIPTUFESVOOFS
    w$ZDMPVEͷϚωʔδυαʔϏε 4BB4

    w໊শ͸(JU)VCIPTUFEΑΓ
    w(JU)VCIPTUFEΑΓ҆Ձʹఏڙ
    wϚϧνςφϯτߏ੒Ͱఏڙ
    wςφϯτͷৄࡉ͸ޙड़
    wෳ਺εϖοΫఏڙ
    wݱࡏ͸-JOVY 6CVOUVGPDBM
    ͷΈ
    w$PSF(#ʙ$PSF(#

    View Slide

  12. $ZDMPVEIPTUFESVOOFS

    View Slide

  13. $ZDMPVEIPTUFESVOOFS
    $ curl -XGET ${your_shoes_host}/target | jq
    .

    [
    {

    "id": "477f6073-90d1-47d8-958f-4707cea61e8d"
    ,

    "scope": "octocat"
    ,

    "token_expired_at": "2006-01-02T15:04:05Z"
    ,

    "ghe_domain": "https://github.example.com"
    ,

    "resource_type": "micro"
    ,

    "runner_user": "ubuntu"
    ,

    "runner_version": ""
    ,

    "provider_url": ""
    ,

    "status": "active"
    ,

    "status_description": ""
    ,

    "created_at": "2006-01-02T15:04:05Z"
    ,

    "updated_at": "2006-01-02T15:04:05Z
    "

    }

    View Slide

  14. $ZDMPVEIPTUFESVOOFS
    $ curl -XGET ${your_shoes_host}/target | jq
    .

    [
    {

    "id": "477f6073-90d1-47d8-958f-4707cea61e8d"
    ,

    "scope": "octocat",
    "token_expired_at": "2006-01-02T15:04:05Z"
    ,

    "ghe_domain": "https://github.example.com"
    ,

    "resource_type": "micro"
    ,

    "runner_user": "ubuntu"
    ,

    "runner_version": ""
    ,

    "provider_url": ""
    ,

    "status": "active"
    ,

    "status_description": ""
    ,

    "created_at": "2006-01-02T15:04:05Z"
    ,

    "updated_at": "2006-01-02T15:04:05Z
    "

    }
    ಈ࡞͢Δ0SHPS3FQPࢦఆ

    View Slide

  15. $ZDMPVEIPTUFESVOOFS
    $ curl -XGET ${your_shoes_host}/target | jq
    .

    [
    {

    "id": "477f6073-90d1-47d8-958f-4707cea61e8d"
    ,

    "scope": "octocat"
    ,

    "token_expired_at": "2006-01-02T15:04:05Z"
    ,

    "ghe_domain": "https://github.example.com",
    "resource_type": "micro"
    ,

    "runner_user": "ubuntu"
    ,

    "runner_version": ""
    ,

    "provider_url": ""
    ,

    "status": "active"
    ,

    "status_description": ""
    ,

    "created_at": "2006-01-02T15:04:05Z"
    ,

    "updated_at": "2006-01-02T15:04:05Z
    "

    }
    (JU)VC&OUFSQSJTF4FSWFSରԠ

    View Slide

  16. $ZDMPVEIPTUFESVOOFS
    $ curl -XGET ${your_shoes_host}/target | jq
    .

    [
    {

    "id": "477f6073-90d1-47d8-958f-4707cea61e8d"
    ,

    "scope": "octocat"
    ,

    "token_expired_at": "2006-01-02T15:04:05Z"
    ,

    "ghe_domain": "https://github.example.com"
    ,

    "resource_type": "micro",
    "runner_user": "ubuntu"
    ,

    "runner_version": ""
    ,

    "provider_url": ""
    ,

    "status": "active"
    ,

    "status_description": ""
    ,

    "created_at": "2006-01-02T15:04:05Z"
    ,

    "updated_at": "2006-01-02T15:04:05Z
    "

    }
    ར༻͢ΔSVOOFSͷεϖοΫ

    ˞0SHSFQP͝ͱʹมߋՄೳ

    View Slide

  17. $ZDMPVEIPTUFESVOOFS
    $ cat test.yaml

    jobs
    :

    test
    :

    runs-on: ubuntu-lates
    t

    steps
    :

    - name: Checkout

    View Slide

  18. $ZDMPVEIPTUFESVOOFS
    $ cat test.yaml

    jobs
    :

    test
    :

    - runs-on: ubuntu-lates
    t

    + runs-on: self-hoste
    d

    steps
    :

    - name: Checkout

    View Slide

  19. $ZDMPVEIPTUFESVOOFS
    ؆୯ʂ🤩

    View Slide

  20. $ZDMPVEIPTUFESVOOFS
    ʹ͓͚Δ
    Ϛϧνςφϯτ

    View Slide

  21. ʮϚϧνςφϯτʯ
    w$ZDMPVEʹ͓͚Δςφϯτ
    wࢠձࣾ
    wҟࣄۀ෦
    w$ZDMPVEIPTUFESVOOFSʹ͓͚Δ

    ςφϯτ
    wϓϩδΣΫτ
    w੥ٻ۠෼
    w(JU)VCυϝΠϯ ()&4

    View Slide

  22. ʮϚϧνςφϯτʯ
    $ZDMPVE5FOBOU"
    $ZDMPVE5FOBOU#
    $ZDMPVEIPTUFESVOOFS5FOBOU"
    $ZDMPVEIPTUFESVOOFS5FOBOU"
    $ZDMPVEIPTUFESVOOFS5FOBOU"
    $ZDMPVEIPTUFESVOOFS5FOBOU#
    $ZDMPVEIPTUFESVOOFS5FOBOU#
    $ZDMPVEIPTUFESVOOFS5FOBOU#

    View Slide

  23. $ZDMPVEIPTUFESVOOFSߏ੒
    Kubernetes Cluster
    myshoes-proxy


    (Global IP)
    github.com


    GHES
    User
    Tenant A
    Tenant B
    webhook
    myshoes REST API
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    k
    8
    s Service
    k
    8
    s Service
    for Service X
    for Service Y
    k
    8
    s Service
    Tenant C
    k
    8
    s Service

    View Slide

  24. $ZDMPVEIPTUFESVOOFSߏ੒
    既存 Cycloud VM
    Tenant A
    Tenant B
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    k
    8
    s Service
    k
    8
    s Service
    for Service X
    for Service Y
    k
    8
    s Service
    Tenant C
    k
    8
    s Service
    Service VM
    Service VM
    Service VM
    既存 Cycloud ユーザは

    ⾃テナントのVMとの接続が可能

    他テナントのVMとの接続は不可

    View Slide

  25. Ϛϧνςφϯτڞ༗Ϩϕϧ
    wશମڞ༗
    w,VCFSOFUFT
    wNZTIPFT͸͡Ίଟϓϩηε
    wNZTIPFTDPOUSPMMFS
    w,VCFSOFUFTDPOUSPMMFS
    w.Z42-
    wXJUIPQFOBSLPSDIFTUSBUPS
    w(JU)VC"QQT

    View Slide

  26. Ϛϧνςφϯτڞ༗Ϩϕϧ
    w$ZDMPVEʹ͓͚Δςφϯτڞ༗
    w4UBEJVNΫϥελ
    w࣮ࡍʹSVOOFS🏃͕࣮ߦ͞ΕΔ7.
    w%PDLFSSFHJTUSZ
    w%PDLFS)VC3BUF-JNJUରࡦ
    w-9%EOTNBTRͰ

    ໊લղܾ݁ՌΛͶ͡ۂ͛ͯӡ༻

    View Slide

  27. Ϛϧνςφϯτڞ༗Ϩϕϧ
    w$ZDMPVEIPTUFESVOOFSʹ͓͚Δ

    ςφϯτڞ༗
    wNZTIPFTίϯςφͷΈ

    View Slide

  28. $ZDMPVEIPTUFESVOOFSߏ੒
    Kubernetes Cluster
    myshoes-proxy


    (Global IP)
    github.com


    GHES
    User
    Tenant A
    Tenant B
    webhook
    myshoes REST API
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    k
    8
    s Service
    k
    8
    s Service
    for Service X
    for Service Y
    k
    8
    s Service
    Tenant C
    k
    8
    s Service

    View Slide

  29. $ZDMPVEIPTUFESVOOFSߏ੒
    Kubernetes Cluster
    myshoes-proxy


    (Global IP)
    github.com


    GHES
    User
    Tenant A
    Tenant B
    webhook
    myshoes REST API
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    k
    8
    s Service
    k
    8
    s Service
    for Service X
    for Service Y
    k
    8
    s Service
    Tenant C
    k
    8
    s Service
    <શମڞ༗>NZTIPFTQSPYZ
    *OUFSOFUˠ1SJWBUF$MPVE
    ֤ςφϯτ΁ৼΓ෼͚
    ೝূೝՄ

    View Slide

  30. $ZDMPVEIPTUFESVOOFSߏ੒
    Kubernetes Cluster
    myshoes-proxy


    (Global IP)
    github.com


    GHES
    User
    Tenant A
    Tenant B
    webhook
    myshoes REST API
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    k
    8
    s Service
    k
    8
    s Service
    for Service X
    for Service Y
    k
    8
    s Service
    Tenant C
    k
    8
    s Service
    <શମڞ༗>

    NZTIPFTDPOUSPMMFS
    5FOBOU $3%
    ͔Βੜ੒
    LT4FSWJDF
    NZTIPFT1PE
    OPEF@TFMFDUPSͰ

    ىಈςφϯτΛ੾ସ

    View Slide

  31. ϚΠΫϩαʔϏεͱͯ͠ӡ༻
    wNZTIPFTDPOUSPMMFS
    wCJMMJOHDPMMFDUPS
    wTIPFTTFSWFS ޙड़

    w.Z42-0QFSBUJPO1PE
    wDFSUNBOBHFS
    wPQFOTUBDLDJOEFSDTJ
    wFUDFUD

    View Slide

  32. Ҿ༻8FCΞϓϦέʔγϣϯΛϚϧνςφϯτܕ4BB4ιϦϡʔγϣϯʹม׵͢Δ
    IUUQTXXXJCNDPNEFWFMPQFSXPSLTKQDMPVEMJCSBSZDMNVMUJUFOBOUTBBT
    Ϛϧνςφϯτͷछྨ

    View Slide

  33. ͳͥ͋͑ͯγϯάϧYςφϯτʁ

    View Slide

  34. ͳͥ͋͑ͯγϯάϧYςφϯτʁ
    NZTIPFTͷ࣮૷ࢦ਑

    View Slide

  35. NZTIPFT 044
    ͷ࣮૷ํ਑
    w؆୯ʹσϓϩΠ
    w/PUOFFE0SDIFTUSBUPS
    w714ࣗ୐αʔό UNVYͰ΋ஔ͚Δಈ͘

    w(PBM/PO(PBMΛখ͘͞໌֬ʹ
    w/PO(PBMͷҰྫ
    w/PNVMUJUFOBODZ
    w/P"VUI

    View Slide

  36. NZTIPFT 044
    ͷ࣮૷ํ਑
    w/PO(PBMͷ໌֬ԽϝϦοτ
    wʮͲ͜Ͱ࣮૷͢΂͖͔ʯ͕໌֬Խ
    wࣾ಺ཁ݅͸ଞ ϓϩΩγ
    ʹ೚ͤΔ
    wϚϧνςφϯτ
    wೝূೝՄ
    wޙ͔Βඞͣ௥Ճཁ݅͸΍ͬͯ͘Δ
    wlͲͷϚΠΫϩαʔϏεʹ௥Ճ͢Δ͔z

    Λߟྀ͢Δඞཁͷͳ͍ઃܭ͕ॏཁ

    View Slide

  37. Ͳ͜ʹԿΛ࣮૷͢΂͖͔
    ڥք͕ϋοΩϦ͢Δ


    ΞϓϦέʔγϣϯ͝ͱͷ

    ੹೚ൣғ͕໌֬Խ💯

    View Slide

  38. 0WFSWJFX͸͜͜·Ͱ

    View Slide

  39. View Slide

  40. #VJME6Qͷ༷ࢠΛ͝঺հ

    View Slide

  41. $ZDMPVEIPTUFESVOOFS
    w೥࢝ຊ֨తʹ։ൃ։࢝
    wύϒϦοΫϕʔλ
    w(FOFSBMMZ"WBJMBCMF
    wςφϯτ਺໿
    w4UBEJVN਺໿7.
    wಉ࣌SVOOFS਺໿ʙ୆

    View Slide

  42. ࠷ॳظ

    View Slide

  43. $ZDMPVEIPTUFESVOOFSߏ੒
    Kubernetes Cluster
    myshoes-proxy


    (Global IP)
    github.com


    GHES
    User
    Tenant A
    Tenant B
    webhook
    myshoes REST API
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs


    LXD (listen
    8
    44 3
    )
    k
    8
    s Service
    k
    8
    s Service
    for Service X
    for Service Y
    k
    8
    s Service
    Tenant C
    k
    8
    s Service

    View Slide

  44. $ZDMPVEIPTUFESVOOFSߏ੒
    Kubernetes Cluster
    myshoes-proxy


    LXD (listen
    8
    44 3
    )
    Stadium VM
    Stadium VM
    Stadium VMs
    8
    s Service
    k
    8
    s Service
    for Service X
    for Service Y
    k
    8

    View Slide

  45. -9%
    w-9$ -JOV9$POUBJOFS
    %BFNPO
    w$BOPOJDBMओಋ
    w(JU)VCIPTUFE 7.
    ʹدͤΔͨΊ࠾༻
    w4FSWJDFDPOUBJOFSར༻Մ
    wLTϕʔεͩͱ%JO%ʹͳͬͯ͠·͏
    w׆ൃʹ։ൃத
    wίϯτϦϏϡʔτ͠·ͨ͠💪
    w৽ػೳ։ൃͳͲ΋׆ൃ

    View Slide

  46. -9%JO1SPEVDUJPO
    w8JLJʹৄࡉͳهࣄ͋Γ
    w1SPEVDUJPO4FUVQ
    w IUUQTMJOVYDPOUBJOFSTPSHMYEEPDTNBTUFSQSPEVDUJPO
    TFUVQ
    w`limits.conf` `sysctl.conf
    `

    w`txqueuelen`ͳͲ
    wݱ࣌఺Ͱ͸

    ͜ͷهࣄͷ௨ΓͰେ͖͘໰୊ແ͠

    View Slide

  47. -9%JO1SPEVDUJPO
    wετϨʔδCUSGT
    w[GT SFDPNNFOE
    %PDLFSಈ͔ͣ
    wEJS$PQZPO8SJUFಈ͔ͣ
    wCUSGTEJTL༻ʹߴ଎/7.F44%ར༻
    w0QFO4UBDLJ4$4*Ͱ7.ʹ઀ଓ
    wੑ্࣭*0͕όʔετʹͳΓ͕ͪ
    wϓϥΠϕʔτΫϥ΢υ͔ͩΒͦ͜

    View Slide

  48. Virtual Machine


    with actions/runner
    Cloud Provider


    • AWS / GCP / Azure


    • LXD / OpenStack


    • etc etc...
    ① Start job
    ② webhook
    ③ CreateInstance
    ④ Create
    ⑤ Register

    View Slide

  49. Virtual Machine


    with actions/runner
    Cloud Provider


    • AWS / GCP / Azure


    • LXD / OpenStack


    • etc etc...
    ① Start job
    ② webhook
    ③ CreateInstance
    ④ Create
    ⑤ Register

    View Slide

  50. NZTIPFTQSPWJEFS
    wΠϯελϯεͷ࡞੒࡟আͷ࣮ߦ૚
    wΫϥ΢υ͝ͱʹॲཧ͕ҟͳΔ
    wಠࣗͷόΠφϦͰந৅Խ

    wIBTIJDPSQHPQMVHJOར༻
    wPT&YFDͰϓϥάΠϯΛىಈ
    wH31$ϕʔεʹωΰγΤʔγϣϯ

    View Slide

  51. ಺෦ಈ࡞NZTIPFTQSPWJEFS
    Cloud Provider
    Execute provider


    (hashicorp/go-plugin)
    Virtual Machine


    with actions/runner
    shoes-xxx
    Virtual Machine


    with actions/runner
    Virtual Machine


    with actions/runner
    Call API
    1
    .Runner Name : runnerに付けるべき名前


    2
    .Setup Script : actions/runner展開スクリプト


    3
    .Resource Type : runnerのスペック

    View Slide

  52. ಺෦ಈ࡞TIPFTMYE
    Execute provider


    (hashicorp/go-plugin)
    shoes-lxd
    Call API
    Stadium
    lxd
    Virtual Machine


    with actions/runner
    Stadium
    lxd
    Virtual Machine


    with actions/runner
    ɾ

    ɾ
    ɾ

    View Slide

  53. TIPFTMYE
    w-9%޲͚NZTIPFTQSPWJEFS
    wૉ๿ʹ-9%"1*Λར༻
    w౰ॳ͸NPDL༻ʹ࣮૷
    wγϯϓϧނʹ͔ͳΓ҆ఆ͍ͯͨ͠
    wlֵ৽తzεέδϡʔϦϯάΞϧΰϦζϜ

    View Slide

  54. TIPFTMYE
    IUUQTHJUIVCDPNXIZXBJUBTIPFTMYECMPCCBFFDDGBCDDBCGDGNBJOHP--

    View Slide

  55. TIPFTMYE
    IUUQTHJUIVCDPNXIZXBJUBTIPFTMYECMPCCBFFDDGBCDDBCGDGNBJOHP--

    View Slide

  56. TIPFTMYE
    ʊਓਓਓਓਓਓʊ
    ʼɹrand.Intnɹʻ
    ʉY^Y^Y^Y^Y^Y^ʉ

    View Slide

  57. TIPFTMYE
    w-9%޲͚NZTIPFTQSPWJEFS
    wૉ๿ʹ-9%"1*Λར༻
    w౰ॳ͸NPDL༻ʹ࣮૷
    wγϯϓϧނʹ͔ͳΓ҆ఆ͍ͯͨ͠
    wlֵ৽తzεέδϡʔϦϯάΞϧΰϦζϜ
    wNBUISBOEͰىಈઌΛܾఆ
    w୹໋ͳόΠφϦͳͷͰେ͖ͳ໰୊ͳ͠

    View Slide

  58. ಺෦ಈ࡞TIPFTMYE
    Execute provider


    (hashicorp/go-plugin)
    shoes-lxd
    Call API
    Stadium
    lxd
    Virtual Machine


    with actions/runner
    Stadium
    lxd
    Virtual Machine


    with actions/runner
    ɾ

    ɾ
    ɾ
    rand.Intn

    View Slide

  59. ෛՙঢ়ଶ

    View Slide

  60. ෛՙঢ়ଶ
    概ね分散

    View Slide

  61. ͔͠͠సػ͕😱

    View Slide

  62. TIPFTMYE҆৺ਆ࿩่յ
    wಥવϢʔβར༻͕૿Ճ
    w4UBEJVN͕଱͖͑Ε่ͣյ🔥
    wϦΫΤετ΁ͷόοϑΝϦϯάࣦഊ
    wṖͷ-9%ϊʔυෆௐ
    wҟৗʹ"1*઀ଓ͕஗Ԇʢ୆ͷΈʣ
    wʮۭ͖ʹ༏ઌͯ͠ೖΕΔʯ͕ࠔ೉
    w࣮૷্ͷແବ΋͋ͬͨ΋ͷͷʜʜ

    View Slide

  63. ෛՙঢ়ଶ

    View Slide

  64. ෛՙঢ়ଶ
    ほぼ0% 🔥🔥🔥

    View Slide

  65. #VJME6Q💪

    View Slide

  66. TIPFTMYENVMUJ։ൃ
    wTIPFTMYEͷͭΒΈ
    wόΠφϦ࣮ߦͳͷͰεςʔτͳ͠
    wͪΐͬͱ͙Β͍͸͍࣋ͪͨʜʜ
    wϥού޲͚σʔϞϯΛ࡞ͬͨ
    wXIZXBJUBTIPFTMYENVMUJ
    wTIPFTMYEͷෳ਺୆൛
    wύϒϦοΫΫϥ΢υͷΑ͏ʹ

    "1*&OEQPJOUΛͭʹ

    View Slide

  67. ಺෦ಈ࡞TIPFTMYE
    Execute provider


    (hashicorp/go-plugin)
    shoes-lxd
    Call API
    Stadium
    lxd
    Virtual Machine


    with actions/runner
    Stadium
    lxd
    Virtual Machine


    with actions/runner
    ɾ

    ɾ
    ɾ

    View Slide

  68. ಺෦ಈ࡞TIPFTMYENVMUJ
    Execute provider


    (hashicorp/go-plugin)
    shoes-lxd-multi
    Call API
    Stadium
    lxd
    Virtual Machine


    with actions/runner
    Stadium
    lxd
    Virtual Machine


    with actions/runner
    ɾ

    ɾ
    ɾ
    shoes-lxd-
    server
    Call (gRPC)

    View Slide

  69. TIPFTMYENVMUJ
    wΠϯελϯεΛੜ੒͢Δίʔυ͸

    TIPFTMYE͔Βίϐϖ
    wαʔόΫϥΠΞϯτϞσϧ΁
    wΩϟύγςΟνΣοΫ
    wNFUSJDTͷ࣮૷
    wར༻͢ΔTUBEJVNӡ༻ͷ؆ૉԽ

    View Slide

  70. ෛՙঢ়ଶ TIPFTMYENVMUJޙ

    View Slide

  71. ෛՙঢ়ଶ TIPFTMYENVMUJޙ

    上限ができた

    View Slide

  72. ऩ༰ϝτϦΫε

    View Slide

  73. ऩ༰ϝτϦΫε
    N倍以上は収容しない

    View Slide

  74. ऩ༰ϝτϦΫε
    他へ順次スケジュール

    View Slide

  75. TIPFTMYENVMUJ։ൃ
    w໾ׂͷ໌֬Խ͕͜͜Ͱ΋Ըܙ
    wεςʔτϨεʹ࣮૷Մೳ
    w৑௕Խ͸ԣʹฒ΂Δ͚ͩ
    wಈ͘Α͏ʹ
    wQSPWJEFSϞσϧʹΑΓҠߦ΋ؾܰ
    wNZTIPFTଆͷίʔυมߋͳ͠
    wςετςφϯτͰͷࢼݧ΋खܰ

    View Slide

  76. ܧଓతͳ#VJME6QͷͨΊͷ

    $*$%ઓज़

    View Slide

  77. ಺෦ಈ࡞TIPFTMYENVMUJ
    Execute provider


    (hashicorp/go-plugin)
    shoes-lxd-multi
    Call API
    Stadium
    lxd
    Virtual Machine


    with actions/runner
    Stadium
    lxd
    Virtual Machine


    with actions/runner
    ɾ

    ɾ
    ɾ
    shoes-lxd-
    server
    Call (gRPC)

    View Slide

  78. ಺෦ಈ࡞TIPFTMYENVMUJ
    Execute provider

    View Slide

  79. ܧଓతͳ#VJME6Q
    wίϯςφ࣮ߦ࣌ʹ֎෦͔Βऔಘ
    w؀ڥม਺PLUGI
    N

    w ./shoes-moc
    k

    w https://example.com/shoes-mock
    w։ൃ؀ڥʹ߹Θͤͯࠩ͠׵͑
    w։ൃ7.ʹ/HJOYೖΕͯόΠφϦ഑৴
    wຊ൪(JU)VC3FMFBTF͔Βऔಘ

    View Slide

  80. ܧଓతͳ#VJME6Q
    apiVersion: myshoes.managed.cycloud.io/v
    1

    kind: Tenant

    spec
    :

    domain: XX
    X

    project: YY
    Y

    plugin_url: https://github.com/whywaita/shoes-lxd-multi/
    releases/download/v0.0.5/shoes-lxd-multi-linux-amd6
    4

    lxd_image_alias: (snip
    )

    myshoes_version: v1.10.7

    environment_values
    :

    MAX_CONCURRENCY_DELETING: '2
    '

    github_apps: (snip)

    lxd_hosts
    :

    - host: https://(snip):8443 # myshoes-stadium-tenantA-00
    1

    - host: https://(snip):8443 # myshoes-stadium-tenantB-002

    View Slide

  81. ܧଓతͳ#VJME6Q
    apiVersion: myshoes.managed.cycloud.io/v
    1

    kind: Tenant

    spec
    :

    domain: XX
    X

    project: YY
    Y

    plugin_url: https://github.com/whywaita/shoes-lxd-multi/
    releases/download/v0.0.5/shoes-lxd-multi-linux-amd6
    4

    lxd_image_alias: (snip
    )

    myshoes_version: v1.10.7

    environment_values
    :

    MAX_CONCURRENCY_DELETING: '2
    '

    github_apps: (snip)

    lxd_hosts
    :

    - host: https://(snip):8443 # myshoes-stadium-tenantA-00
    1

    - host: https://(snip):8443 # myshoes-stadium-tenantB-002

    View Slide

  82. ܧଓతͳ#VJME6Q
    apiVersion: myshoes.managed.cycloud.io/v
    1

    kind: Tenant

    spec
    :

    domain: XX
    X

    project: YY
    Y

    plugin_url: http://10.X.Y.Z/shoes-lxd-multi-linux-amd6
    4

    lxd_image_alias: (snip
    )

    myshoes_version: v1.10.7

    environment_values
    :

    MAX_CONCURRENCY_DELETING: '2
    '

    github_apps: (snip)

    lxd_hosts
    :

    - host: https://(snip):8443 # myshoes-stadium-tenantA-00
    1

    - host: https://(snip):8443 # myshoes-stadium-tenantB-002

    View Slide

  83. ܧଓతͳ#VJME6Q
    w֤ίϛοτຖʹJNBHFΛϏϧυ
    w(JU)VC"DUJPOTΛར༻
    w ()&$ ()&4ͱ΋ʹ
    wϦϦʔείϛοτ͚ͩͰ͸ͳ͘શͯ
    wಛఆςφϯτͷΈCFUB൛ެ։
    IUUQTHJUIVCDPNXIZXBJUBNZTIPFTQLHTDPOUBJOFSNZTIPFT

    View Slide

  84. ܧଓతͳ#VJME6Q
    apiVersion: myshoes.managed.cycloud.io/v
    1

    kind: Tenant

    spec
    :

    domain: XX
    X

    project: YY
    Y

    plugin_url: https://github.com/whywaita/shoes-lxd-multi/
    releases/download/v0.0.5/shoes-lxd-multi-linux-amd6
    4

    lxd_image_alias: (snip
    )

    myshoes_version: v1.10.7

    environment_values
    :

    MAX_CONCURRENCY_DELETING: '2
    '

    github_apps: (snip)

    lxd_hosts
    :

    - host: https://(snip):8443 # myshoes-stadium-tenantA-00
    1

    - host: https://(snip):8443 # myshoes-stadium-tenantB-002

    View Slide

  85. ܧଓతͳ#VJME6Q
    apiVersion: myshoes.managed.cycloud.io/v
    1

    kind: Tenant

    spec
    :

    domain: XX
    X

    project: YY
    Y

    plugin_url: https://github.com/whywaita/shoes-lxd-multi/
    releases/download/v0.0.5/shoes-lxd-multi-linux-amd6
    4

    lxd_image_alias: (snip
    )

    myshoes_version: sha-xxxxxx

    environment_values
    :

    MAX_CONCURRENCY_DELETING: '2
    '

    github_apps: (snip)

    lxd_hosts
    :

    - host: https://(snip):8443 # myshoes-stadium-tenantA-00
    1

    - host: https://(snip):8443 # myshoes-stadium-tenantB-002

    View Slide

  86. ܧଓతͳ#VJME6Q

    View Slide

  87. ܧଓతͳ#VJME6Q
    apiVersion: myshoes.managed.cycloud.io/v
    1

    kind: Tenant

    spec
    :

    domain: XX
    X

    project: YY
    Y

    plugin_url: https://github.com/whywaita/shoes-lxd-multi/
    releases/download/v0.0.5/shoes-lxd-multi-linux-amd6
    4

    lxd_image_alias: (snip
    )

    myshoes_version: v1.10.7

    environment_values
    :

    MAX_CONCURRENCY_DELETING: '2
    '

    github_apps: (snip)

    lxd_hosts
    :

    - host: https://(snip):8443 # myshoes-stadium-tenantA-00
    1

    - host: https://(snip):8443 # myshoes-stadium-tenantB-002

    View Slide

  88. ܧଓతͳ#VJME6Q
    wSVOOFSΠϝʔδ΋"MXBZT#VJME
    wXIZXBJUBWJSUVBMFOWJSPONFOUTMYE
    wຖ೔-9%޲͚ΠϝʔδͷϏϧυ
    wมԽͷܹ͍͠਌ϦϙδτϦΛଊ͑Δ
    wఆظతͳΠϝʔδߋ৽
    w্هࣾ಺ઃఆͰఆظϏϧυ
    wదٓ৽͍͠όʔδϣϯΛσϓϩΠ

    View Slide

  89. ܧଓతͳ#VJME6Q
    wϢʔβ૿ՃʹΑΓ4UBEJVN૿ઃ

    ˠͳ͔ͥδϣϒ࣮ߦ͕஗Ԇ😱
    w࣮૷ʹෆඋ͕͋ͬͯ௚ྻಈ࡞ͩͬͨ
    wՄೳͳݶΓฒྻԽ
    w-9%(P4%,ʹDPOUFYUରԠΛ

    $POUSJCVUF MYDMYE

    View Slide

  90. ܧଓతͳ#VJME6QͷͨΊ
    ͷӡ༻

    View Slide

  91. ؂ࢹ1SPNFUIFVT
    wશମతʹGET /metricsͰ౷Ұ
    wNZTIPFT ֤छϚΠΫϩαʔϏε
    w$ZDMPVE7.্ͷ1SPNFUIFVT
    w%BTICPBSEΛॱ࣍ڞ༗
    w໰୊ղܾ࣌ͷڞ༗ʹར༻
    wো֐ൃੜىҼΛॱ࣍ϝτϦΫεʹ௥Ճ<>
    <>$ZCFS"HFOUͷϓϥΠϕʔτΫϥ΢υ$ZDMPVEͷӡ༻ٴͼϞχλϦϯάʹ͍ͭͯ$0%5
    IUUQTXXXTMJEFTIBSFOFUXIZXBJUBDZCFSBHFOUDZDMPVEBENJOJTUSBUJPOBOENPOJUPSJOHGPSDZDMPVE

    View Slide

  92. ؂ࢹϝτϦΫε Ұྫ

    wNZTIPFT
    w(JU)VC"1*ͷ3BUF-JNJU
    wฒྻ࡞੒ɾ࡟আ਺
    wTIPFTMYETFSWFS
    w4UBEJVNͷར༻཰
    wಈ࡞͍ͯ͠ΔΠϯελϯεͷϦετ

    View Slide

  93. Πϯελϯε਺͔Βݟ͑Δۀ຿࣌ؒ

    View Slide

  94. ӡ༻ͷ೔ʑ

    View Slide

  95. ӡ༻ͷ೔ʑ

    View Slide

  96. ӡ༻ͷ೔ʑ
    対応を実施

    View Slide

  97. ϩάΛਖ਼͘͠औಘ
    wႢಹతͳδϣϒͷ؂ࢹ
    wޙ௥͍ͯ͠ਖ਼͘͠໰୊Λଊ͑Δඞཁ
    w-9%ίϯςφϩάDSPOSTZOD
    w$SPOKPCPO,VCFSOFUFT
    wCBDLVQNZTRM
    wEFMFUFMPHT
    wΞϓϦϩάͷ؂ࢹHPPHMFNUBJM

    View Slide

  98. ਖ਼͘͠ӡ༻։ൃ͢Δ͜ͱͰ
    ਖ਼͘͠ܧଓతͳ#VJME6Q

    View Slide

  99. ͦͯ͠

    View Slide

  100. 5PCFDPOUJOVFE
    IUUQTHJUIVCDPNXIZXBJUBTIPFTBHFOU

    View Slide

  101. ·ͱΊ

    View Slide

  102. ·ͱΊ
    w$ZDMPVEIPTUFESVOOFSͷ֓ཁ
    wNZTIPFT
    wNZTIPFTQSPWJEFSʹ͍ͭͯ
    wϚϧνςφϯτ
    wNZTIPFTͰͷऔΓ૊Έ
    wܧଓతͳ#VJME6Qʹ͍ͭͯ
    wΞϓϦέʔγϣϯઃܭ
    wӡ༻͠΍͍͢ΞϓϦέʔγϣϯ։ൃ

    View Slide