Upgrade to Pro — share decks privately, control downloads, hide ads and more …

深入GitLabCI-談描述檔重構與流水線加速

Ab628671841774343b1020f22f712069?s=47 mouson
November 28, 2020

 深入GitLabCI-談描述檔重構與流水線加速

本次的講題是在 「台中自由軟體愛好者社群」所做的分享,內容主要著重在 GitLab CI 在流水線加速與描述檔重構兩個部分。

使用之 .gitlab-ci.yml 範例連結:https://gitlab.com/mouson-gitlab-playground/gitlab-ci-demo-sample/-/tree/refactor-gitlab-ci-sample-v3

影片紀錄連結:https://www.youtube.com/watch?v=8wLGw7oYJKg&feature=youtu.be&t=2438

Ab628671841774343b1020f22f712069?s=128

mouson

November 28, 2020
Tweet

Transcript

  1. 深入 GitLab CI 談描述檔重構與流⽔線加速 墨嗓 (陳佑⽵) 2020/11/28 @ 台中⾃由軟體愛好者社群

  2. About Me • 陳佑⽵,朋友都叫我墨嗓(Mouson)。 專注於資訊科技整合、應⽤、開發⽅ 法及研究的T型⼈。 妞妞

  3. 歡迎加入 GitLab Hero

  4. . 社群 . https://fb.com/groups/GitLabTaipei https://t.me/GitLabTaiwan

  5. 進入主題

  6. 深入 GitLab CI • Part I: 流⽔線 (Pipeline) 加速 •

    Part II: .gitlab-ci.yml 描述檔如何重構
  7. 怎麼讓流⽔線 (Pipeline) 更快⼀點? Part I

  8. 流⽔線加速三要素

  9. 流⽔線加速三要素 • ⼯作 少做點

  10. 流⽔線加速三要素 • ⼯作 少做點 • 事情 作快點

  11. 流⽔線加速三要素 • ⼯作 少做點 • 事情 作快點 • 作事的 多⼀點

  12. 流⽔線 Pipeline 啟動到結束 哪些地⽅花了時間?

  13. 拆解

  14. .gitlab-ci.yml

  15. .gitlab-ci.yml

  16. .gitlab-ci.yml

  17. .gitlab-ci.yml Job Job Job Job Job Job Job Queue

  18. .gitlab-ci.yml Job Job Job Job Job Job Job Queue Runner

    Server Runner Server Runners Runner Runner Runner Runner
  19. .gitlab-ci.yml Job Job Job Job Job Job Job Queue Runner

    Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________
  20. .gitlab-ci.yml Job Job Job Job Job Job Job Queue

  21. •什麼⼯作先作比較好? .gitlab-ci.yml Job Job Job Job Job Job Job Queue

  22. 什麼⼯作先做比較好?

  23. 什麼⼯作先做比較好? • 跑的快的⼯作

  24. 什麼⼯作先做比較好? • 跑的快的⼯作 • 早點做完早點確認結果

  25. 什麼⼯作先做比較好? • 跑的快的⼯作 • 早點做完早點確認結果 • 有決定性的⼯作

  26. 什麼⼯作先做比較好? • 跑的快的⼯作 • 早點做完早點確認結果 • 有決定性的⼯作 • 沒通過直接 Reject

  27. •什麼⼯作先做比較好? .gitlab-ci.yml Job Job Job Job Job Job Job Queue

  28. •什麼⼯作先做比較好? •每個⼯作都是該做的嗎? .gitlab-ci.yml Job Job Job Job Job Job Job

    Queue
  29. 每個⼯作都是該做的嗎?

  30. 每個⼯作都是該做的嗎? • 同⼀專案中包含前端及後端的測試

  31. 每個⼯作都是該做的嗎? • 同⼀專案中包含前端及後端的測試 • 可以考慮對應原始碼變更才執⾏對應的測試

  32. 每個⼯作都是該做的嗎? • 同⼀專案中包含前端及後端的測試 • 可以考慮對應原始碼變更才執⾏對應的測試

  33. •什麼⼯作先做比較好? •每個⼯作都是該做的嗎? .gitlab-ci.yml Job Job Job Job Job Job Job

    Queue
  34. •什麼⼯作先做比較好? •每個⼯作都是該做的嗎? •有沒有⼯作花太久時間等 待進 Queue ? .gitlab-ci.yml Job Job Job

    Job Job Job Job Queue
  35. ⼯作花太久時間等待進 Queue?

  36. ⼯作花太久時間等待進 Queue? build test deploy build_a test_a deploy_a build_b test_b

    deploy_b build_c test_c deploy_c
  37. ⼯作花太久時間等待進 Queue? build test deploy build_a test_a deploy_a build_b test_b

    deploy_b build_c test_c deploy_c build_b
  38. ⼯作花太久時間等待進 Queue? build test deploy build_a test_a deploy_a build_b test_b

    deploy_b build_c test_c deploy_c build_b
  39. ⼯作花太久時間等待進 Queue? • 套⽤ DAG 機制 (GitLab 12.2) build test

    deploy build_a test_a deploy_a build_b test_b deploy_b build_c test_c deploy_c build_b
  40. ⼯作花太久時間等待進 Queue? • 套⽤ DAG 機制 (GitLab 12.2) build test

    deploy build_a test_a deploy_a build_b test_b deploy_b build_c test_c deploy_c build_b
  41. ⼯作花太久時間等待進 Queue? • 套⽤ DAG 機制 (GitLab 12.2) build test

    deploy build_a test_a deploy_a build_b test_b deploy_b build_c test_c deploy_c build_b
  42. .gitlab-ci.yml Job Job Job Job Job Job Job Queue

  43. .gitlab-ci.yml Job Job Job Job Job Job Job Queue Runner

    Server Runner Server Runners Runner Runner Runner Runner
  44. Job Job Job Job Job Job Job Queue Runner Server

    Runner Server Runners Runner Runner Runner Runner
  45. Job Job Job Job Job Job Job Queue Runner Server

    Runner Server Runners Runner Runner Runner Runner •Runner 數量夠嗎?
  46. Job Job Job Job Job Job Job Queue Runner Server

    Runner Server Runners Runner Runner Runner Runner •Runner 數量夠嗎? •能執⾏的 Runner 夠嗎?
  47. Job Job Job Job Job Job Job Queue Runner Server

    Runner Server Runners Runner Runner Runner Runner
  48. Job Job Job Job Job Job Job Queue Runner Server

    Runner Server Runners Runner Runner Runner Runner Run Job ____________________________
  49. Runner Server Runner Server Runners Runner Runner Runner Runner Run

    Job ____________________________
  50. Runner Server Runner Server Runners Runner Runner Runner Runner Run

    R u n ____________________________
  51. Runner Server Runner Server Runners Runner Runner Runner Runner Run

    R u n ____________________________ * prepare p r e p a r e p r e
  52. Runner Server Runner Server Runners Runner Runner Runner Runner Run

    R u n ____________________________ * prepare p r e p a r e p r e * git g i t g i t r
  53. Runner Server Runner Server Runners Runner Runner Runner Runner Run

    R u n ____________________________ * prepare p r e p a r e p r e * git g i t g i t r * restore r e s t o
  54. Runner Server Runner Server Runners Runner Runner Runner Runner Run

    R u n ____________________________ * prepare p r e p a r e p r e * git g i t g i t r * restore r e s t o * download d o w n l o a d
  55. Runner Server Runner Server Runners Runner Runner Runner Runner Run

    R u n ____________________________ * prepare p r e p a r e p r e * git g i t g i t r * restore r e s t o * download d o w n l o a d * script s c r i p t
  56. Runner Server Runner Server Runners Runner Runner Runner Runner Run

    R u n ____________________________ * prepare p r e p a r e p r e * git g i t g i t r * restore r e s t o * download d o w n l o a d * script s c r i p t * package p a c k a g e
  57. Runner Server Runner Server Runners Runner Runner Runner Runner Run

    R u n ____________________________ * prepare p r e p a r e p r e * git g i t g i t r * restore r e s t o * download d o w n l o a d * script s c r i p t * package p a c k a g e * saving s a v i n
  58. Runner Server Runner Server Runners Runner Runner Runner Runner Run

    R u n ____________________________ * prepare p r e p a r e p r e * git g i t g i t r * restore r e s t o * download d o w n l o a d * script s c r i p t * package p a c k a g e * saving s a v i n * uploading u p l o a d i n g
  59. Runner Server Runner Server Runners Runner Runner Runner Runner Run

    Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up f i le base var
  60. •Network Performance ? Runner Server Runner Server Runners Runner Runner

    Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up f i le base var
  61. •Network Performance ? •Docker image size ? Runner Server Runner

    Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execut e * package install * saving cache * uploading artifacts * cleaning up f i le base var
  62. Docker Image Size ?

  63. Docker Image Size ?

  64. •Network Performance ? •Docker image size ? •Git Repo Size

    ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execut e * package install * saving cache * uploading artifacts * cleaning up f i le base var
  65. •Network Performance ? •Docker image size ? •Git Repo Size

    ? •Programming Language Dependence Package Size ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execut e * package install * saving cache * uploading artifacts * cleaning up f i le base var
  66. •Network Performance ? •Docker image size ? •Git Repo Size

    ? •Programming Language Dependence Package Size ? •Artifacts Size ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execut e * package install * saving cache * uploading artifacts * cleaning up f i le base var
  67. •Network Performance ? •Docker image size ? •Git Repo Size

    ? •Programming Language Dependence Package Size ? •Artifacts Size ? •Script Performance ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execut e * package install * saving cache * uploading artifacts * cleaning up f i le base var
  68. Runner Server Runner Server Runners Runner Runner Runner Runner Run

    Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execut e * package install * saving cache * uploading artifacts * cleaning up f i le base var
  69. .gitlab-ci.yml Job Job Job Job Job Job Job Queue Runner

    Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execut e * package install * saving cache * uploading artifacts * cleaning up f i le base var
  70. Part I Q&A

  71. .gitlab-ci.yml 描述檔 如何重構? Part II

  72. 從前從前.... 使⽤ GitLab CI ⾃動檢查格式

  73. 加入⾃動測試

  74. 加入⾃動部署

  75. 再增加 ⼀種版本...

  76. 越來越難以維護的 .gitlab-ci.yml 該怎麼整理?

  77. 怎麼整理? 針對 v73, v74 分別有: lint、test、deploy 三種類型的⼯作

  78. 1. 擷取⼀致的 部分到 Default: 擷取⼀致的 script 變為 default:

  79. 1. 擷取⼀致的 部分到 Default: 擷取⼀致的 script 變為 default:

  80. Default: 1. Pipeline 中所有⼯作都會套⽤ 2. 如不符使⽤,可以覆寫 3. ⽀援:image、services、 before_script、after_script、tags、 cache、artifacts、retry、timeout、

    interruptible
  81. 2. 整理 script 共同之程序 整理出來的 Script,可以做程序的分 組,⽅便後續使⽤

  82. 2. 整理 script 共同之程序 整理出來的 Script,可以做程序的分 組,⽅便後續使⽤

  83. 2. 整理 script 共同之程序 整理出來的 Script,可以做程序的分 組,⽅便後續使⽤

  84. 3. 擷取重複段 落以繼承再利⽤ 利⽤ hidden job 整理 job 共同的部分 並使⽤

    extends 繼承
  85. 3. 擷取重複段 落以繼承再利⽤ 利⽤ hidden job 整理 job 共同的部分 並使⽤

    extends 繼承
  86. 3. 擷取重複段 落以繼承再利⽤ 利⽤ hidden job 整理 job 共同的部分 並使⽤

    extends 繼承
  87. hidden job 1. 以 「.」開頭的⼯作名稱: 整個⼯作都不執⾏

  88. extends:

  89. extends: 1. 把繼承來源的內容抄⼀次

  90. extends: 1. 把繼承來源的內容抄⼀次 2. 可以允許既成多個來源

  91. extends: 1. 把繼承來源的內容抄⼀次 2. 可以允許既成多個來源 3. 重複的內容:

  92. extends: 1. 把繼承來源的內容抄⼀次 2. 可以允許既成多個來源 3. 重複的內容: script: 後蓋前

  93. extends: 1. 把繼承來源的內容抄⼀次 2. 可以允許既成多個來源 3. 重複的內容: script: 後蓋前 variables:

    取合集後蓋前
  94. 4.利⽤Anchors 整理可複⽤的 script

  95. 4.利⽤Anchors 整理可複⽤的 script

  96. 4.利⽤Anchors 整理可複⽤的 script

  97. Anchors 1. 可允許多⾏ 2. 可在 script 中使⽤

  98. Anchors 1. 可允許多⾏ 2. 可在 script 中使⽤

  99. Anchors 1. 可允許多⾏ 2. 可在 script 中使⽤

  100. 5. 建立同類⼯ 作模板 tmpl

  101. 5. 建立同類⼯ 作模板 tmpl

  102. 5. 建立同類⼯ 作模板 tmpl extends: 可以允許多層次繼承

  103. 6. 擷取為 變數控制

  104. 6. 擷取為 變數控制

  105. 6. 擷取為 變數控制

  106. 6. 擷取為 變數控制

  107. 7. 整理⾄ Template 中

  108. 7. 整理⾄ Template 中

  109. 7. 整理⾄ Template 中

  110. 8. 使⽤ Parallel Matrix

  111. 8. 使⽤ Parallel Matrix

  112. 8. 使⽤ Parallel Matrix

  113. 8. 使⽤ Parallel Matrix

  114. 9. 擷取 Parallel Matrix 為模板供 繼承

  115. 9. 擷取 Parallel Matrix 為模板供 繼承

  116. 9. 擷取 Parallel Matrix 為模板供 繼承 entends: 允許既成多個來源

  117. 10. 拆分檔案利 ⽤ include 載入

  118. 10. 拆分檔案利 ⽤ include 載入

  119. 10. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├──

    common.yml ├── de f i ne.yml ├── lint.yml ├── test.yml └── deploy.yml
  120. 10. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├──

    common.yml ├── de f i ne.yml ├── lint.yml ├── test.yml └── deploy.yml
  121. 10. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├──

    common.yml ├── de f i ne.yml ├── lint.yml ├── test.yml └── deploy.yml
  122. 10. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├──

    common.yml ├── de f i ne.yml ├── lint.yml ├── test.yml └── deploy.yml
  123. 10. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├──

    common.yml ├── de f i ne.yml ├── lint.yml ├── test.yml └── deploy.yml
  124. 10. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├──

    common.yml ├── de f i ne.yml ├── lint.yml ├── test.yml └── deploy.yml
  125. 10. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├──

    common.yml ├── de f i ne.yml ├── lint.yml ├── test.yml └── deploy.yml
  126. 10. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├──

    common.yml ├── de f i ne.yml ├── lint.yml ├── test.yml └── deploy.yml
  127. include: - local: 同專案下的檔案 - remote: URL 連結 - template:

    GitLab 官⽅提供的模板 - f i le: 來⾃其他專案的檔案
  128. 11. 建立共⽤流 程供其他專案 使⽤

  129. include: f i le introduced in GitLab 11.7 include multi

    f i le from a project in GitLab 13.6 11. 建立共⽤流 程供其他專案 使⽤
  130. 變更及覆寫參數 11. 建立共⽤流 程供其他專案 使⽤

  131. 變更及覆寫參數 11. 建立共⽤流 程供其他專案 使⽤

  132. 12. 從模板選 擇需要的⼯作

  133. 12. 從模板選 擇需要的⼯作

  134. Recap • 利⽤ default stage 整理所有 job 都共同的 • 利⽤

    .hidden 建立模板⼯作,並⽤ extends 繼承使⽤ • 利⽤ Anchors 整理可重複使⽤的 Script • 萃取出變數 • 利⽤ Parallel Matrix 功能 • 建立多專案共⽤流程 • Sample Source Code: https://bit.ly/3mTiBgJ
  135. Sample Code • https://bit.ly/3mTiBgJ

  136. 歡迎加入 GitLab Hero

  137. Q&A