Upgrade to Pro — share decks privately, control downloads, hide ads and more …

深入GitLabCI-談描述檔重構與流水線加速

 深入GitLabCI-談描述檔重構與流水線加速

本次的講題是在 「台中自由軟體愛好者社群」所做的分享,內容主要著重在 GitLab CI 在流水線加速與描述檔重構兩個部分。

使用之 .gitlab-ci.yml 範例連結:https://gitlab.com/mouson-gitlab-playground/gitlab-ci-demo-sample/-/tree/refactor-gitlab-ci-sample-v3

影片紀錄連結:https://www.youtube.com/watch?v=8wLGw7oYJKg&feature=youtu.be&t=2438

mouson(墨嗓)

November 28, 2020
Tweet

More Decks by mouson(墨嗓)

Other Decks in Technology

Transcript

  1. 深入 GitLab CI


    談描述檔重構與流⽔線加速
    墨嗓 (陳佑⽵)


    2020/11/28 @ 台中⾃由軟體愛好者社群

    View Slide

  2. About Me
    • 陳佑⽵,朋友都叫我墨嗓(Mouson)。
    專注於資訊科技整合、應⽤、開發⽅
    法及研究的T型⼈。


    妞妞

    View Slide

  3. 歡迎加入 GitLab Hero

    View Slide

  4. .
    社群
    .
    https://fb.com/groups/GitLabTaipei https://t.me/GitLabTaiwan

    View Slide

  5. 進入主題

    View Slide

  6. 深入 GitLab CI
    • Part I: 流⽔線 (Pipeline) 加速


    • Part II: .gitlab-ci.yml 描述檔如何重構

    View Slide

  7. 怎麼讓流⽔線 (Pipeline)


    更快⼀點?
    Part I

    View Slide

  8. 流⽔線加速三要素

    View Slide

  9. 流⽔線加速三要素
    • ⼯作 少做點

    View Slide

  10. 流⽔線加速三要素
    • ⼯作 少做點
    • 事情 作快點

    View Slide

  11. 流⽔線加速三要素
    • ⼯作 少做點
    • 事情 作快點
    • 作事的 多⼀點

    View Slide

  12. 流⽔線 Pipeline 啟動到結束


    哪些地⽅花了時間?

    View Slide

  13. 拆解

    View Slide

  14. .gitlab-ci.yml

    View Slide

  15. .gitlab-ci.yml

    View Slide

  16. .gitlab-ci.yml

    View Slide

  17. .gitlab-ci.yml Job
    Job
    Job
    Job
    Job
    Job
    Job Queue

    View Slide

  18. .gitlab-ci.yml Job
    Job
    Job
    Job
    Job
    Job
    Job Queue
    Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner

    View Slide

  19. .gitlab-ci.yml Job
    Job
    Job
    Job
    Job
    Job
    Job Queue
    Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner
    Run Job


    ____________________________


    View Slide

  20. .gitlab-ci.yml Job
    Job
    Job
    Job
    Job
    Job
    Job Queue

    View Slide

  21. •什麼⼯作先作比較好?
    .gitlab-ci.yml Job
    Job
    Job
    Job
    Job
    Job
    Job Queue

    View Slide

  22. 什麼⼯作先做比較好?

    View Slide

  23. 什麼⼯作先做比較好?
    • 跑的快的⼯作

    View Slide

  24. 什麼⼯作先做比較好?
    • 跑的快的⼯作
    • 早點做完早點確認結果

    View Slide

  25. 什麼⼯作先做比較好?
    • 跑的快的⼯作
    • 早點做完早點確認結果
    • 有決定性的⼯作

    View Slide

  26. 什麼⼯作先做比較好?
    • 跑的快的⼯作
    • 早點做完早點確認結果
    • 有決定性的⼯作
    • 沒通過直接 Reject

    View Slide

  27. •什麼⼯作先做比較好?
    .gitlab-ci.yml Job
    Job
    Job
    Job
    Job
    Job
    Job Queue

    View Slide

  28. •什麼⼯作先做比較好?
    •每個⼯作都是該做的嗎?
    .gitlab-ci.yml Job
    Job
    Job
    Job
    Job
    Job
    Job Queue

    View Slide

  29. 每個⼯作都是該做的嗎?

    View Slide

  30. 每個⼯作都是該做的嗎?
    • 同⼀專案中包含前端及後端的測試

    View Slide

  31. 每個⼯作都是該做的嗎?
    • 同⼀專案中包含前端及後端的測試
    • 可以考慮對應原始碼變更才執⾏對應的測試

    View Slide

  32. 每個⼯作都是該做的嗎?
    • 同⼀專案中包含前端及後端的測試
    • 可以考慮對應原始碼變更才執⾏對應的測試

    View Slide

  33. •什麼⼯作先做比較好?
    •每個⼯作都是該做的嗎?
    .gitlab-ci.yml Job
    Job
    Job
    Job
    Job
    Job
    Job Queue

    View Slide

  34. •什麼⼯作先做比較好?
    •每個⼯作都是該做的嗎?
    •有沒有⼯作花太久時間等
    待進 Queue ?
    .gitlab-ci.yml Job
    Job
    Job
    Job
    Job
    Job
    Job Queue

    View Slide

  35. ⼯作花太久時間等待進 Queue?

    View Slide

  36. ⼯作花太久時間等待進 Queue?
    build test deploy
    build_a test_a deploy_a
    build_b test_b deploy_b
    build_c test_c deploy_c

    View Slide

  37. ⼯作花太久時間等待進 Queue?
    build test deploy
    build_a test_a deploy_a
    build_b test_b deploy_b
    build_c test_c deploy_c
    build_b

    View Slide

  38. ⼯作花太久時間等待進 Queue?
    build test deploy
    build_a test_a deploy_a
    build_b test_b deploy_b
    build_c test_c deploy_c
    build_b

    View Slide

  39. ⼯作花太久時間等待進 Queue?
    • 套⽤ DAG 機制 (GitLab 12.2)
    build test deploy
    build_a test_a deploy_a
    build_b test_b deploy_b
    build_c test_c deploy_c
    build_b

    View Slide

  40. ⼯作花太久時間等待進 Queue?
    • 套⽤ DAG 機制 (GitLab 12.2)
    build test deploy
    build_a test_a deploy_a
    build_b test_b deploy_b
    build_c test_c deploy_c
    build_b

    View Slide

  41. ⼯作花太久時間等待進 Queue?
    • 套⽤ DAG 機制 (GitLab 12.2)
    build test deploy
    build_a test_a deploy_a
    build_b test_b deploy_b
    build_c test_c deploy_c
    build_b

    View Slide

  42. .gitlab-ci.yml Job
    Job
    Job
    Job
    Job
    Job
    Job Queue

    View Slide

  43. .gitlab-ci.yml Job
    Job
    Job
    Job
    Job
    Job
    Job Queue
    Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner

    View Slide

  44. Job
    Job
    Job
    Job
    Job
    Job
    Job Queue
    Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner

    View Slide

  45. Job
    Job
    Job
    Job
    Job
    Job
    Job Queue
    Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner
    •Runner 數量夠嗎?

    View Slide

  46. Job
    Job
    Job
    Job
    Job
    Job
    Job Queue
    Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner
    •Runner 數量夠嗎?
    •能執⾏的 Runner 夠嗎?

    View Slide

  47. Job
    Job
    Job
    Job
    Job
    Job
    Job Queue
    Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner

    View Slide

  48. Job
    Job
    Job
    Job
    Job
    Job
    Job Queue
    Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner
    Run Job


    ____________________________


    View Slide

  49. Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner
    Run Job


    ____________________________


    View Slide

  50. Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner
    Run
    R
    u
    n
    ____________________________

    View Slide

  51. Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner
    Run
    R
    u
    n
    ____________________________
    * prepare
    p
    r
    e
    p
    a
    r
    e
    p
    r
    e

    View Slide

  52. Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner
    Run
    R
    u
    n
    ____________________________
    * prepare
    p
    r
    e
    p
    a
    r
    e
    p
    r
    e
    * git
    g
    i
    t
    g
    i
    t
    r

    View Slide

  53. Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner
    Run
    R
    u
    n
    ____________________________
    * prepare
    p
    r
    e
    p
    a
    r
    e
    p
    r
    e
    * git
    g
    i
    t
    g
    i
    t
    r
    * restore
    r
    e
    s
    t
    o

    View Slide

  54. Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner
    Run
    R
    u
    n
    ____________________________
    * prepare
    p
    r
    e
    p
    a
    r
    e
    p
    r
    e
    * git
    g
    i
    t
    g
    i
    t
    r
    * restore
    r
    e
    s
    t
    o
    * download
    d
    o
    w
    n
    l
    o
    a
    d

    View Slide

  55. Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner
    Run
    R
    u
    n
    ____________________________
    * prepare
    p
    r
    e
    p
    a
    r
    e
    p
    r
    e
    * git
    g
    i
    t
    g
    i
    t
    r
    * restore
    r
    e
    s
    t
    o
    * download
    d
    o
    w
    n
    l
    o
    a
    d
    * script
    s
    c
    r
    i
    p
    t

    View Slide

  56. Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner
    Run
    R
    u
    n
    ____________________________
    * prepare
    p
    r
    e
    p
    a
    r
    e
    p
    r
    e
    * git
    g
    i
    t
    g
    i
    t
    r
    * restore
    r
    e
    s
    t
    o
    * download
    d
    o
    w
    n
    l
    o
    a
    d
    * script
    s
    c
    r
    i
    p
    t
    * package
    p
    a
    c
    k
    a
    g
    e

    View Slide

  57. Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner
    Run
    R
    u
    n
    ____________________________
    * prepare
    p
    r
    e
    p
    a
    r
    e
    p
    r
    e
    * git
    g
    i
    t
    g
    i
    t
    r
    * restore
    r
    e
    s
    t
    o
    * download
    d
    o
    w
    n
    l
    o
    a
    d
    * script
    s
    c
    r
    i
    p
    t
    * package
    p
    a
    c
    k
    a
    g
    e
    * saving
    s
    a
    v
    i
    n

    View Slide

  58. Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner
    Run
    R
    u
    n
    ____________________________
    * prepare
    p
    r
    e
    p
    a
    r
    e
    p
    r
    e
    * git
    g
    i
    t
    g
    i
    t
    r
    * restore
    r
    e
    s
    t
    o
    * download
    d
    o
    w
    n
    l
    o
    a
    d
    * script
    s
    c
    r
    i
    p
    t
    * package
    p
    a
    c
    k
    a
    g
    e
    * saving
    s
    a
    v
    i
    n
    * uploading
    u
    p
    l
    o
    a
    d
    i
    n
    g

    View Slide

  59. Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner
    Run Job
    ____________________________
    * prepare execute env
    * git repo fetch
    * restore cache
    * download artifacts
    * script execute
    * package install
    * saving cache
    * uploading artifacts
    * cleaning up
    f
    i
    le base var

    View Slide

  60. •Network Performance ?
    Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner
    Run Job
    ____________________________
    * prepare execute env
    * git repo fetch
    * restore cache
    * download artifacts
    * script execute
    * package install
    * saving cache
    * uploading artifacts
    * cleaning up
    f
    i
    le base var

    View Slide

  61. •Network Performance ?


    •Docker image size ?
    Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner
    Run Job


    ____________________________


    * prepare execute env


    * git repo fetch


    * restore cache


    * download artifacts


    * script execut
    e


    * package install


    * saving cache


    * uploading artifacts


    * cleaning up
    f
    i
    le base var

    View Slide

  62. Docker Image Size ?

    View Slide

  63. Docker Image Size ?

    View Slide

  64. •Network Performance ?


    •Docker image size ?


    •Git Repo Size ?
    Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner
    Run Job


    ____________________________


    * prepare execute env


    * git repo fetch


    * restore cache


    * download artifacts


    * script execut
    e


    * package install


    * saving cache


    * uploading artifacts


    * cleaning up
    f
    i
    le base var

    View Slide

  65. •Network Performance ?


    •Docker image size ?


    •Git Repo Size ?


    •Programming Language
    Dependence Package
    Size ?
    Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner
    Run Job


    ____________________________


    * prepare execute env


    * git repo fetch


    * restore cache


    * download artifacts


    * script execut
    e


    * package install


    * saving cache


    * uploading artifacts


    * cleaning up
    f
    i
    le base var

    View Slide

  66. •Network Performance ?


    •Docker image size ?


    •Git Repo Size ?


    •Programming Language
    Dependence Package
    Size ?


    •Artifacts Size ?
    Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner
    Run Job


    ____________________________


    * prepare execute env


    * git repo fetch


    * restore cache


    * download artifacts


    * script execut
    e


    * package install


    * saving cache


    * uploading artifacts


    * cleaning up
    f
    i
    le base var

    View Slide

  67. •Network Performance ?


    •Docker image size ?


    •Git Repo Size ?


    •Programming Language
    Dependence Package
    Size ?


    •Artifacts Size ?


    •Script Performance ?
    Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner
    Run Job


    ____________________________


    * prepare execute env


    * git repo fetch


    * restore cache


    * download artifacts


    * script execut
    e


    * package install


    * saving cache


    * uploading artifacts


    * cleaning up
    f
    i
    le base var

    View Slide

  68. Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner
    Run Job


    ____________________________


    * prepare execute env


    * git repo fetch


    * restore cache


    * download artifacts


    * script execut
    e


    * package install


    * saving cache


    * uploading artifacts


    * cleaning up
    f
    i
    le base var

    View Slide

  69. .gitlab-ci.yml Job
    Job
    Job
    Job
    Job
    Job
    Job Queue
    Runner


    Server
    Runner


    Server
    Runners
    Runner
    Runner
    Runner
    Runner
    Run Job


    ____________________________


    * prepare execute env


    * git repo fetch


    * restore cache


    * download artifacts


    * script execut
    e


    * package install


    * saving cache


    * uploading artifacts


    * cleaning up
    f
    i
    le base var

    View Slide

  70. Part I


    Q&A

    View Slide

  71. .gitlab-ci.yml 描述檔


    如何重構?
    Part II

    View Slide

  72. 從前從前....
    使⽤ GitLab CI ⾃動檢查格式

    View Slide

  73. 加入⾃動測試

    View Slide

  74. 加入⾃動部署

    View Slide

  75. 再增加


    ⼀種版本...

    View Slide

  76. 越來越難以維護的


    .gitlab-ci.yml
    該怎麼整理?

    View Slide

  77. 怎麼整理?
    針對 v73, v74 分別有:


    lint、test、deploy 三種類型的⼯作

    View Slide

  78. 1. 擷取⼀致的
    部分到 Default:
    擷取⼀致的 script 變為 default:

    View Slide

  79. 1. 擷取⼀致的
    部分到 Default:
    擷取⼀致的 script 變為 default:

    View Slide

  80. Default:
    1. Pipeline 中所有⼯作都會套⽤


    2. 如不符使⽤,可以覆寫


    3. ⽀援:image、services、
    before_script、after_script、tags、
    cache、artifacts、retry、timeout、
    interruptible

    View Slide

  81. 2. 整理 script
    共同之程序
    整理出來的 Script,可以做程序的分
    組,⽅便後續使⽤

    View Slide

  82. 2. 整理 script
    共同之程序
    整理出來的 Script,可以做程序的分
    組,⽅便後續使⽤

    View Slide

  83. 2. 整理 script
    共同之程序
    整理出來的 Script,可以做程序的分
    組,⽅便後續使⽤

    View Slide

  84. 3. 擷取重複段
    落以繼承再利⽤
    利⽤ hidden job


    整理 job 共同的部分


    並使⽤ extends 繼承

    View Slide

  85. 3. 擷取重複段
    落以繼承再利⽤
    利⽤ hidden job


    整理 job 共同的部分


    並使⽤ extends 繼承

    View Slide

  86. 3. 擷取重複段
    落以繼承再利⽤
    利⽤ hidden job


    整理 job 共同的部分


    並使⽤ extends 繼承

    View Slide

  87. hidden job
    1. 以 「.」開頭的⼯作名稱:


    整個⼯作都不執⾏

    View Slide

  88. extends:

    View Slide

  89. extends:
    1. 把繼承來源的內容抄⼀次

    View Slide

  90. extends:
    1. 把繼承來源的內容抄⼀次
    2. 可以允許既成多個來源

    View Slide

  91. extends:
    1. 把繼承來源的內容抄⼀次
    2. 可以允許既成多個來源
    3. 重複的內容:

    View Slide

  92. extends:
    1. 把繼承來源的內容抄⼀次
    2. 可以允許既成多個來源
    3. 重複的內容:
    script: 後蓋前

    View Slide

  93. extends:
    1. 把繼承來源的內容抄⼀次
    2. 可以允許既成多個來源
    3. 重複的內容:
    script: 後蓋前
    variables: 取合集後蓋前

    View Slide

  94. 4.利⽤Anchors


    整理可複⽤的
    script

    View Slide

  95. 4.利⽤Anchors


    整理可複⽤的
    script

    View Slide

  96. 4.利⽤Anchors


    整理可複⽤的
    script

    View Slide

  97. Anchors
    1. 可允許多⾏


    2. 可在 script 中使⽤

    View Slide

  98. Anchors
    1. 可允許多⾏


    2. 可在 script 中使⽤

    View Slide

  99. Anchors
    1. 可允許多⾏


    2. 可在 script 中使⽤

    View Slide

  100. 5. 建立同類⼯
    作模板 tmpl

    View Slide

  101. 5. 建立同類⼯
    作模板 tmpl

    View Slide

  102. 5. 建立同類⼯
    作模板 tmpl
    extends: 可以允許多層次繼承

    View Slide

  103. 6. 擷取為


    變數控制

    View Slide

  104. 6. 擷取為


    變數控制

    View Slide

  105. 6. 擷取為


    變數控制

    View Slide

  106. 6. 擷取為


    變數控制

    View Slide

  107. 7. 整理⾄
    Template 中

    View Slide

  108. 7. 整理⾄
    Template 中

    View Slide

  109. 7. 整理⾄
    Template 中

    View Slide

  110. 8. 使⽤
    Parallel Matrix

    View Slide

  111. 8. 使⽤
    Parallel Matrix

    View Slide

  112. 8. 使⽤
    Parallel Matrix

    View Slide

  113. 8. 使⽤
    Parallel Matrix

    View Slide

  114. 9. 擷取 Parallel
    Matrix 為模板供
    繼承

    View Slide

  115. 9. 擷取 Parallel
    Matrix 為模板供
    繼承

    View Slide

  116. 9. 擷取 Parallel
    Matrix 為模板供
    繼承
    entends: 允許既成多個來源

    View Slide

  117. 10. 拆分檔案利
    ⽤ include 載入

    View Slide

  118. 10. 拆分檔案利
    ⽤ include 載入

    View Slide

  119. 10. 拆分檔案利
    ⽤ include 載入
    ├── .gitlab-ci.yml


    └── templates


    ├── common.yml


    ├── de
    f
    i
    ne.yml


    ├── lint.yml


    ├── test.yml


    └── deploy.yml


    View Slide

  120. 10. 拆分檔案利
    ⽤ include 載入
    ├── .gitlab-ci.yml


    └── templates


    ├── common.yml


    ├── de
    f
    i
    ne.yml


    ├── lint.yml


    ├── test.yml


    └── deploy.yml


    View Slide

  121. 10. 拆分檔案利
    ⽤ include 載入
    ├── .gitlab-ci.yml


    └── templates


    ├── common.yml


    ├── de
    f
    i
    ne.yml


    ├── lint.yml


    ├── test.yml


    └── deploy.yml


    View Slide

  122. 10. 拆分檔案利
    ⽤ include 載入
    ├── .gitlab-ci.yml


    └── templates


    ├── common.yml


    ├── de
    f
    i
    ne.yml


    ├── lint.yml


    ├── test.yml


    └── deploy.yml


    View Slide

  123. 10. 拆分檔案利
    ⽤ include 載入
    ├── .gitlab-ci.yml


    └── templates


    ├── common.yml


    ├── de
    f
    i
    ne.yml


    ├── lint.yml


    ├── test.yml


    └── deploy.yml


    View Slide

  124. 10. 拆分檔案利
    ⽤ include 載入
    ├── .gitlab-ci.yml


    └── templates


    ├── common.yml


    ├── de
    f
    i
    ne.yml


    ├── lint.yml


    ├── test.yml


    └── deploy.yml


    View Slide

  125. 10. 拆分檔案利
    ⽤ include 載入
    ├── .gitlab-ci.yml


    └── templates


    ├── common.yml


    ├── de
    f
    i
    ne.yml


    ├── lint.yml


    ├── test.yml


    └── deploy.yml


    View Slide

  126. 10. 拆分檔案利
    ⽤ include 載入
    ├── .gitlab-ci.yml


    └── templates


    ├── common.yml


    ├── de
    f
    i
    ne.yml


    ├── lint.yml


    ├── test.yml


    └── deploy.yml


    View Slide

  127. include:
    - local: 同專案下的檔案


    - remote: URL 連結


    - template: GitLab 官⽅提供的模板


    -
    f
    i
    le: 來⾃其他專案的檔案


    View Slide

  128. 11. 建立共⽤流
    程供其他專案


    使⽤

    View Slide

  129. include:
    f
    i
    le introduced in GitLab 11.7


    include multi
    f
    i
    le from a project in
    GitLab 13.6
    11. 建立共⽤流
    程供其他專案


    使⽤

    View Slide

  130. 變更及覆寫參數
    11. 建立共⽤流
    程供其他專案


    使⽤

    View Slide

  131. 變更及覆寫參數
    11. 建立共⽤流
    程供其他專案


    使⽤

    View Slide

  132. 12. 從模板選
    擇需要的⼯作

    View Slide

  133. 12. 從模板選
    擇需要的⼯作

    View Slide

  134. Recap
    • 利⽤ default stage 整理所有 job 都共同的


    • 利⽤ .hidden 建立模板⼯作,並⽤ extends 繼承使⽤


    • 利⽤ Anchors 整理可重複使⽤的 Script


    • 萃取出變數


    • 利⽤ Parallel Matrix 功能


    • 建立多專案共⽤流程


    • Sample Source Code: https://bit.ly/3mTiBgJ

    View Slide

  135. Sample Code
    • https://bit.ly/3mTiBgJ

    View Slide

  136. 歡迎加入 GitLab Hero

    View Slide

  137. Q&A

    View Slide