Save 37% off PRO during our Black Friday Sale! »

Advanced GitLab Workshop - 談 pipeline 調教與進階 GitLab CI 語法

Ab628671841774343b1020f22f712069?s=47 mouson
November 24, 2021

Advanced GitLab Workshop - 談 pipeline 調教與進階 GitLab CI 語法

此為 DevOpsDays Taipei 2021 中的 Workshop Advanced GitLab CI 所使用的簡報,與簡報搭配的範例 GitLab Repo:
https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24

PS. 本次工作坊使用的範例及對應步驟,大部分都可以在對應的 GitLab 中找到對應的 Git 分支。可透過 GitLab pipeline editor 直接選擇分支進行查看及編輯。

Ab628671841774343b1020f22f712069?s=128

mouson

November 24, 2021
Tweet

Transcript

  1. Advanced GitLab CI 談 pipeline 調教與進階 GitLab CI 墨嗓 (陳佑⽵)

    2021/11/24 @ DevOpsDays Taipei 2021
  2. About Me • 陳佑⽵,朋友都叫我墨嗓(Mouson)。 專注於資訊科技整合、應⽤、開發⽅ 法及研究的T型⼈。 妞妞

  3. None
  4. DevOpsDaysGitLab

  5. 本⽇練習範例 https://bit.ly/DevOpsDaysGitLab https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24

  6. Fork https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24 https://bit.ly/DevOpsDaysGitLab

  7. 到⾃⼰的空間 https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24 https://bit.ly/DevOpsDaysGitLab

  8. 皆已建立分⽀ https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24 https://bit.ly/DevOpsDaysGitLab

  9. 進入 Pipeline https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24 https://bit.ly/DevOpsDaysGitLab

  10. Run Pipeline https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24 https://bit.ly/DevOpsDaysGitLab

  11. Run Pipeline https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24 https://bit.ly/DevOpsDaysGitLab

  12. 使⽤ Pipeline Editor https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24 https://bit.ly/DevOpsDaysGitLab

  13. Part I:GitLab CI 進階語法 Part II:加速 GitLab CI Pipeline

  14. Part I:GitLab CI 進階語法 Part II:加速 GitLab CI Pipeline

  15. 從前從前.... 使⽤ GitLab CI ⾃動檢查格式

  16. 加入⾃動測試

  17. 加入⾃動部署

  18. 再增加 ⼀種版本... 分⽀:00-init

  19. 越來越難以維護的 .gitlab-ci.yml 該怎麼整理?

  20. 怎麼整理? 針對 v73, v74 分別有: lint、test、deploy 三種類型的⼯作

  21. 01. 擷取⼀致的 部分到 Default: 擷取⼀致的 script 變為 default: 分⽀:01-default

  22. 01. 擷取⼀致的 部分到 Default: 擷取⼀致的 script 變為 default: 分⽀:01-default

  23. Default: 1. Pipeline 中所有⼯作都會套⽤ 2. 如不符使⽤,可以覆寫 3. ⽀援:image、services、 before_script、after_script、tags、 cache、artifacts、retry、timeout、

    interruptible 分⽀:01-default
  24. Default: 不想沿⽤ Default,可以使⽤ inherit:default https://docs.gitlab.com/ee/ci/yaml/index.html#inheritdefault

  25. 02. 整理 script 共同之程序 整理出來的 Script,可以做程序的分 組,⽅便後續使⽤ 分⽀:02-extract-to-before-script

  26. 2. 整理 script 共同之程序 整理出來的 Script,可以做程序的分 組,⽅便後續使⽤ 分⽀:02-extract-to-before-script

  27. 02. 整理 script 共同之程序 整理出來的 Script,可以做程序的分 組,⽅便後續使⽤ 分⽀:02-extract-to-before-script

  28. 03. 擷取重複段 落以繼承再利⽤ 利⽤ hidden job 整理 job 共同的部分 並使⽤

    extends 繼承 分⽀:03-extract-before-script-t...
  29. 03. 擷取重複段 落以繼承再利⽤ 利⽤ hidden job 整理 job 共同的部分 並使⽤

    extends 繼承 分⽀:03-extract-before-script-t...
  30. 03. 擷取重複段 落以繼承再利⽤ 利⽤ hidden job 整理 job 共同的部分 並使⽤

    extends 繼承 分⽀:03-extract-before-script-t...
  31. hidden job 1. 以 「.」開頭的⼯作名稱: 整個⼯作都不執⾏ 分⽀:03-extract-before-script-t...

  32. extends: 分⽀:03-extract-before-script-t...

  33. extends: 1. 把繼承來源的內容抄⼀次 分⽀:03-extract-before-script-t...

  34. extends: 1. 把繼承來源的內容抄⼀次 2. 可以允許既成多個來源 分⽀:03-extract-before-script-t...

  35. extends: 1. 把繼承來源的內容抄⼀次 2. 可以允許既成多個來源 3. 重複的內容: 分⽀:03-extract-before-script-t...

  36. extends: 1. 把繼承來源的內容抄⼀次 2. 可以允許既成多個來源 3. 重複的內容: script: 後蓋前 分⽀:03-extract-before-script-t...

  37. extends: 1. 把繼承來源的內容抄⼀次 2. 可以允許既成多個來源 3. 重複的內容: script: 後蓋前 variables:

    取合集後蓋前 分⽀:03-extract-before-script-t...
  38. 04.利⽤Anchors 整理可複⽤的 script 分⽀:04-extract-prepare-process

  39. 04.利⽤Anchors 整理可複⽤的 script 分⽀:04-extract-prepare-process

  40. 04.利⽤Anchors 整理可複⽤的 script 分⽀:04-extract-prepare-process

  41. Anchors 1. 可允許多⾏ 2. 可在 script 中使⽤ 分⽀:04-extract-prepare-process

  42. Anchors 1. 可允許多⾏ 2. 可在 script 中使⽤ 分⽀:04-extract-prepare-process

  43. Anchors 1. 可允許多⾏ 2. 可在 script 中使⽤ 分⽀:04-extract-prepare-process

  44. 04-02. 利⽤ !reference 整理可複⽤的 script (GitLab 13.9) 04-02-extract-prepare-process https://docs.gitlab.com/ee/ci/yaml/yaml_optimization.html

  45. !reference vs. anchor tags !reference 可以參考 include 的 yaml 04-02-extract-prepare-process

    https://docs.gitlab.com/ee/ci/yaml/yaml_optimization.html
  46. 05-1. 擷取為 變數控制 05-1-extract-to-variables

  47. 05-1. 擷取為 變數控制 05-1-extract-to-variables

  48. 05-1. 擷取為 變數控制 05-1-extract-to-variables

  49. 05-2. 建立同類 ⼯作模板 tmpl extends: 可以允許多層次繼承 05-2-extract-job-to-template

  50. 05-2. 建立同類 ⼯作模板 tmpl extends: 可以允許多層次繼承 05-2-extract-job-to-template

  51. extends: 可以允許多層次繼承 06. 使⽤ Parallel Matrix 06-extract-to-parallel-matrix

  52. 06. 使⽤ Parallel Matrix 06-extract-to-parallel-matrix

  53. 06. 使⽤ Parallel Matrix 06-extract-to-parallel-matrix

  54. 07. 擷取 Parallel Matrix 為模板供 繼承 07-extract-version-de fi ne-to-....

  55. 07. 擷取 Parallel Matrix 為模板供 繼承 07-extract-version-de fi ne-to-....

  56. 07. 擷取 Parallel Matrix 為模板供 繼承 entends: 允許既成多個來源 07-extract-version-de fi

    ne-to-....
  57. 08. 拆分檔案利 ⽤ include 載入 08-extract-to-template-yaml

  58. 08. 拆分檔案利 ⽤ include 載入 08-extract-to-template-yaml

  59. 08. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├──

    common.yml ├── de fi ne.yml ├── lint.yml ├── test.yml └── deploy.yml 08-extract-to-template-yaml
  60. 08-extract-to-template-yaml 08. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates

    ├── common.yml ├── de fi ne.yml ├── lint.yml ├── test.yml └── deploy.yml
  61. 08-extract-to-template-yaml 08. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates

    ├── common.yml ├── de fi ne.yml ├── lint.yml ├── test.yml └── deploy.yml
  62. 08-extract-to-template-yaml 08. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates

    ├── common.yml ├── de fi ne.yml ├── lint.yml ├── test.yml └── deploy.yml
  63. 08-extract-to-template-yaml 08. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates

    ├── common.yml ├── de fi ne.yml ├── lint.yml ├── test.yml └── deploy.yml
  64. 08-extract-to-template-yaml 08. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates

    ├── common.yml ├── de fi ne.yml ├── lint.yml ├── test.yml └── deploy.yml
  65. 08-extract-to-template-yaml 08. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates

    ├── common.yml ├── de fi ne.yml ├── lint.yml ├── test.yml └── deploy.yml
  66. 08. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├──

    common.yml ├── de fi ne.yml ├── lint.yml ├── test.yml └── deploy.yml 08-extract-to-template-yaml
  67. include: - local: 同專案下的檔案 - remote: URL 連結 - template:

    GitLab 官⽅提供的模板 - fi le: 來⾃其他專案的檔案
  68. include: - local: 同專案下的檔案 - remote: URL 連結 - template:

    GitLab 官⽅提供的模板 - fi le: 來⾃其他專案的檔案 GitLab 13.11.
  69. 09. 建立共⽤流 程供其他專案 使⽤ 09-used-template-from-other-project

  70. include: fi le introduced in GitLab 11.7 include multi fi

    le from a project in GitLab 13.6 09. 建立共⽤流 程供其他專案 使⽤ 09-used-template-from-other-project
  71. 變更及覆寫參數 09. 建立共⽤流 程供其他專案 使⽤ 09-used-template-from-other-project

  72. 變更及覆寫參數 09-1. 變數變 更及覆寫 09-1-overwrite-job-and-variable

  73. 09-2. 從模板選 擇需要的⼯作

  74. 09-2. 從模板選 擇需要的⼯作

  75. Recap • 利⽤ default stage 整理所有 job 都共同的 • 利⽤

    .hidden 建立模板⼯作,並⽤ extends 繼承使⽤ • 利⽤ Anchors、Reference 整理可重複使⽤的 Script • 萃取出變數 • 利⽤ Parallel Matrix 功能 • 建立多專案共⽤流程
  76. Part I:GitLab CI 進階語法 Part II:加速 GitLab CI Pipeline

  77. 流⽔線加速三要素

  78. 流⽔線加速三要素 • ⼯作 少做點

  79. 流⽔線加速三要素 • ⼯作 少做點 • 事情 作快點

  80. 流⽔線加速三要素 • ⼯作 少做點 • 事情 作快點 • 作事的 多⼀點

  81. 流⽔線 Pipeline 啟動到結束 哪些地⽅花了時間?

  82. .gitlab-ci.yml

  83. .gitlab-ci.yml

  84. .gitlab-ci.yml

  85. .gitlab-ci.yml Job Job Job Job Job Job Job Queue

  86. .gitlab-ci.yml Job Job Job Job Job Job Job Queue Runner

    Server Runner Server Runners Runner Runner Runner Runner
  87. .gitlab-ci.yml Job Job Job Job Job Job Job Queue Runner

    Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________
  88. .gitlab-ci.yml Job Job Job Job Job Job Job Queue

  89. •什麼⼯作先作比較好? .gitlab-ci.yml Job Job Job Job Job Job Job Queue

  90. 什麼⼯作先做比較好?

  91. 什麼⼯作先做比較好? • 跑的快的⼯作

  92. 什麼⼯作先做比較好? • 跑的快的⼯作 • 早點做完早點確認結果

  93. 什麼⼯作先做比較好? • 跑的快的⼯作 • 早點做完早點確認結果 • 有決定性的⼯作

  94. 什麼⼯作先做比較好? • 跑的快的⼯作 • 早點做完早點確認結果 • 有決定性的⼯作 • 沒通過直接 Reject

  95. •什麼⼯作先做比較好? .gitlab-ci.yml Job Job Job Job Job Job Job Queue

  96. •什麼⼯作先做比較好? •每個⼯作都是該做的嗎? .gitlab-ci.yml Job Job Job Job Job Job Job

    Queue
  97. 每個⼯作都是該做的嗎?

  98. 每個⼯作都是該做的嗎? • 同⼀專案中包含前端及後端的測試

  99. 每個⼯作都是該做的嗎? • 同⼀專案中包含前端及後端的測試 • 可以考慮對應原始碼變更才執⾏對應的測試

  100. 每個⼯作都是該做的嗎? • 同⼀專案中包含前端及後端的測試 • 可以考慮對應原始碼變更才執⾏對應的測試

  101. •什麼⼯作先做比較好? •每個⼯作都是該做的嗎? .gitlab-ci.yml Job Job Job Job Job Job Job

    Queue
  102. •什麼⼯作先做比較好? •每個⼯作都是該做的嗎? •有沒有⼯作花太久時間等 待進 Queue ? .gitlab-ci.yml Job Job Job

    Job Job Job Job Queue
  103. ⼯作花太久時間等待進 Queue? 分⽀:10-stage-without-dag-needs

  104. ⼯作花太久時間等待進 Queue? build test deploy build_a test_a deploy_a build_b test_b

    deploy_b build_c test_c deploy_c 分⽀:10-stage-without-dag-needs
  105. ⼯作花太久時間等待進 Queue? build test deploy build_a test_a deploy_a build_b test_b

    deploy_b build_c test_c deploy_c build_b 分⽀:10-stage-without-dag-needs
  106. ⼯作花太久時間等待進 Queue? build test deploy build_a test_a deploy_a build_b test_b

    deploy_b build_c test_c deploy_c build_b 分⽀:10-stage-without-dag-needs
  107. ⼯作花太久時間等待進 Queue? • 套⽤ DAG 機制 (GitLab 12.2) build test

    deploy build_a test_a deploy_a build_b test_b deploy_b build_c test_c deploy_c build_b 分⽀:10-1-stage-with-needs
  108. ⼯作花太久時間等待進 Queue? • 套⽤ DAG 機制 (GitLab 12.2) build test

    deploy build_a test_a deploy_a build_b test_b deploy_b build_c test_c deploy_c build_b 分⽀:10-1-stage-with-needs
  109. ⼯作花太久時間等待進 Queue? • 套⽤ DAG 機制 (GitLab 12.2) build test

    deploy build_a test_a deploy_a build_b test_b deploy_b build_c test_c deploy_c build_b 分⽀:10-1-stage-with-needs
  110. .gitlab-ci.yml Job Job Job Job Job Job Job Queue

  111. .gitlab-ci.yml Job Job Job Job Job Job Job Queue Runner

    Server Runner Server Runners Runner Runner Runner Runner
  112. Job Job Job Job Job Job Job Queue Runner Server

    Runner Server Runners Runner Runner Runner Runner
  113. Job Job Job Job Job Job Job Queue Runner Server

    Runner Server Runners Runner Runner Runner Runner •Runner 數量夠嗎?
  114. Job Job Job Job Job Job Job Queue Runner Server

    Runner Server Runners Runner Runner Runner Runner •Runner 數量夠嗎? •能執⾏的 Runner 夠嗎?
  115. Job Job Job Job Job Job Job Queue Runner Server

    Runner Server Runners Runner Runner Runner Runner
  116. Job Job Job Job Job Job Job Queue Runner Server

    Runner Server Runners Runner Runner Runner Runner Run Job ____________________________
  117. Runner Server Runner Server Runners Runner Runner Runner Runner Run

    Job ____________________________
  118. Runner Server Runner Server Runners Runner Runner Runner Runner Run

    Run ____________________________
  119. Runner Server Runner Server Runners Runner Runner Runner Runner Run

    Run ____________________________ * prepare p r e pa r e p r e
  120. Runner Server Runner Server Runners Runner Runner Runner Runner Run

    Run ____________________________ * prepare p r e pa r e p r e * git g i t gi t r
  121. Runner Server Runner Server Runners Runner Runner Runner Runner Run

    Run ____________________________ * prepare p r e pa r e p r e * git g i t gi t r * restore re s t o
  122. Runner Server Runner Server Runners Runner Runner Runner Runner Run

    Run ____________________________ * prepare p r e pa r e p r e * git g i t gi t r * restore re s t o * download do wnlo ad
  123. Runner Server Runner Server Runners Runner Runner Runner Runner Run

    Run ____________________________ * prepare p r e pa r e p r e * git g i t gi t r * restore re s t o * download do wnlo ad * script s c r i p t
  124. Runner Server Runner Server Runners Runner Runner Runner Runner Run

    Run ____________________________ * prepare p r e pa r e p r e * git g i t gi t r * restore re s t o * download do wnlo ad * script s c r i p t * package p ack a ge
  125. Runner Server Runner Server Runners Runner Runner Runner Runner Run

    Run ____________________________ * prepare p r e pa r e p r e * git g i t gi t r * restore re s t o * download do wnlo ad * script s c r i p t * package p ack a ge * saving s a vin
  126. Runner Server Runner Server Runners Runner Runner Runner Runner Run

    Run ____________________________ * prepare p r e pa r e p r e * git g i t gi t r * restore re s t o * download do wnlo ad * script s c r i p t * package p ack a ge * saving s a vin * uploading u p load i ng
  127. Runner Server Runner Server Runners Runner Runner Runner Runner Run

    Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var
  128. •Network Performance ? Runner Server Runner Server Runners Runner Runner

    Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var
  129. •Network Performance ? •Docker image size ? Runner Server Runner

    Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var
  130. Docker Image Size ? 分⽀:11-compare-difference-image

  131. •Network Performance ? •Docker image size ? •Git Repo Size

    ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var
  132. •Network Performance ? •Docker image size ? •Git Repo Size

    ? •Programming Language Dependence Package Size ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var
  133. •Network Performance ? •Docker image size ? •Git Repo Size

    ? •Programming Language Dependence Package Size ? •Artifacts Size ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var
  134. •Network Performance ? •Docker image size ? •Git Repo Size

    ? •Programming Language Dependence Package Size ? •Artifacts Size ? •Script Performance ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var
  135. Runner Server Runner Server Runners Runner Runner Runner Runner Run

    Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var
  136. .gitlab-ci.yml Job Job Job Job Job Job Job Queue Runner

    Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var
  137. Q&A