Advanced GitLab Workshop - 談 pipeline 調教與進階 GitLab CI 語法

此為 DevOpsDays Taipei 2021 中的 Workshop Advanced GitLab CI 所使用的簡報，與簡報搭配的範例 GitLab Repo：
https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24

PS. 本次工作坊使用的範例及對應步驟，大部分都可以在對應的 GitLab 中找到對應的 Git 分支。可透過 GitLab pipeline editor 直接選擇分支進行查看及編輯。

Transcript

1. Advanced GitLab CI 談 pipeline 調教與進階 GitLab CI 墨嗓 (陳佑⽵) 2021/11/24 @ DevOpsDays Taipei 2021
2. About Me • 陳佑⽵，朋友都叫我墨嗓(Mouson)。 專注於資訊科技整合、應⽤、開發⽅ 法及研究的T型⼈。 妞妞
4. DevOpsDaysGitLab
5. 本⽇練習範例 https://bit.ly/DevOpsDaysGitLab https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24
6. Fork https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24 https://bit.ly/DevOpsDaysGitLab
7. 到⾃⼰的空間 https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24 https://bit.ly/DevOpsDaysGitLab
8. 皆已建立分⽀ https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24 https://bit.ly/DevOpsDaysGitLab
9. 進入 Pipeline https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24 https://bit.ly/DevOpsDaysGitLab
10. Run Pipeline https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24 https://bit.ly/DevOpsDaysGitLab
11. Run Pipeline https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24 https://bit.ly/DevOpsDaysGitLab
12. 使⽤ Pipeline Editor https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24 https://bit.ly/DevOpsDaysGitLab
13. Part I：GitLab CI 進階語法 Part II：加速 GitLab CI Pipeline
14. Part I：GitLab CI 進階語法 Part II：加速 GitLab CI Pipeline
15. 從前從前.... 使⽤ GitLab CI ⾃動檢查格式
16. 加入⾃動測試
17. 加入⾃動部署
18. 再增加 ⼀種版本... 分⽀：00-init
19. 越來越難以維護的 .gitlab-ci.yml 該怎麼整理？
20. 怎麼整理？ 針對 v73, v74 分別有： lint、test、deploy 三種類型的⼯作
21. 01. 擷取⼀致的 部分到 Default: 擷取⼀致的 script 變為 default: 分⽀：01-default
22. 01. 擷取⼀致的 部分到 Default: 擷取⼀致的 script 變為 default: 分⽀：01-default
23. Default: 1. Pipeline 中所有⼯作都會套⽤ 2. 如不符使⽤，可以覆寫 3. ⽀援：image、services、 before_script、after_script、tags、 cache、artifacts、retry、timeout、 interruptible 分⽀：01-default
24. Default: 不想沿⽤ Default，可以使⽤ inherit:default https://docs.gitlab.com/ee/ci/yaml/index.html#inheritdefault
25. 02. 整理 script 共同之程序 整理出來的 Script，可以做程序的分 組，⽅便後續使⽤ 分⽀：02-extract-to-before-script
26. 2. 整理 script 共同之程序 整理出來的 Script，可以做程序的分 組，⽅便後續使⽤ 分⽀：02-extract-to-before-script
27. 02. 整理 script 共同之程序 整理出來的 Script，可以做程序的分 組，⽅便後續使⽤ 分⽀：02-extract-to-before-script
28. 03. 擷取重複段 落以繼承再利⽤ 利⽤ hidden job 整理 job 共同的部分 並使⽤ extends 繼承 分⽀：03-extract-before-script-t...
29. 03. 擷取重複段 落以繼承再利⽤ 利⽤ hidden job 整理 job 共同的部分 並使⽤ extends 繼承 分⽀：03-extract-before-script-t...
30. 03. 擷取重複段 落以繼承再利⽤ 利⽤ hidden job 整理 job 共同的部分 並使⽤ extends 繼承 分⽀：03-extract-before-script-t...
31. hidden job 1. 以 「.」開頭的⼯作名稱： 整個⼯作都不執⾏ 分⽀：03-extract-before-script-t...
32. extends: 分⽀：03-extract-before-script-t...
33. extends: 1. 把繼承來源的內容抄⼀次 分⽀：03-extract-before-script-t...
34. extends: 1. 把繼承來源的內容抄⼀次 2. 可以允許既成多個來源 分⽀：03-extract-before-script-t...
35. extends: 1. 把繼承來源的內容抄⼀次 2. 可以允許既成多個來源 3. 重複的內容： 分⽀：03-extract-before-script-t...
36. extends: 1. 把繼承來源的內容抄⼀次 2. 可以允許既成多個來源 3. 重複的內容： script: 後蓋前 分⽀：03-extract-before-script-t...
37. extends: 1. 把繼承來源的內容抄⼀次 2. 可以允許既成多個來源 3. 重複的內容： script: 後蓋前 variables: 取合集後蓋前 分⽀：03-extract-before-script-t...
38. 04.利⽤Anchors 整理可複⽤的 script 分⽀：04-extract-prepare-process
39. 04.利⽤Anchors 整理可複⽤的 script 分⽀：04-extract-prepare-process
40. 04.利⽤Anchors 整理可複⽤的 script 分⽀：04-extract-prepare-process
41. Anchors 1. 可允許多⾏ 2. 可在 script 中使⽤ 分⽀：04-extract-prepare-process
42. Anchors 1. 可允許多⾏ 2. 可在 script 中使⽤ 分⽀：04-extract-prepare-process
43. Anchors 1. 可允許多⾏ 2. 可在 script 中使⽤ 分⽀：04-extract-prepare-process
44. 04-02. 利⽤ !reference 整理可複⽤的 script (GitLab 13.9) 04-02-extract-prepare-process https://docs.gitlab.com/ee/ci/yaml/yaml_optimization.html
45. !reference vs. anchor tags !reference 可以參考 include 的 yaml 04-02-extract-prepare-process https://docs.gitlab.com/ee/ci/yaml/yaml_optimization.html
46. 05-1. 擷取為 變數控制 05-1-extract-to-variables
47. 05-1. 擷取為 變數控制 05-1-extract-to-variables
48. 05-1. 擷取為 變數控制 05-1-extract-to-variables
49. 05-2. 建立同類 ⼯作模板 tmpl extends: 可以允許多層次繼承 05-2-extract-job-to-template
50. 05-2. 建立同類 ⼯作模板 tmpl extends: 可以允許多層次繼承 05-2-extract-job-to-template
51. extends: 可以允許多層次繼承 06. 使⽤ Parallel Matrix 06-extract-to-parallel-matrix
52. 06. 使⽤ Parallel Matrix 06-extract-to-parallel-matrix
53. 06. 使⽤ Parallel Matrix 06-extract-to-parallel-matrix
54. 07. 擷取 Parallel Matrix 為模板供 繼承 07-extract-version-de fi ne-to-....
55. 07. 擷取 Parallel Matrix 為模板供 繼承 07-extract-version-de fi ne-to-....
56. 07. 擷取 Parallel Matrix 為模板供 繼承 entends: 允許既成多個來源 07-extract-version-de fi ne-to-....
57. 08. 拆分檔案利 ⽤ include 載入 08-extract-to-template-yaml
58. 08. 拆分檔案利 ⽤ include 載入 08-extract-to-template-yaml
59. 08. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├── common.yml ├── de fi ne.yml ├── lint.yml ├── test.yml └── deploy.yml 08-extract-to-template-yaml
60. 08-extract-to-template-yaml 08. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├── common.yml ├── de fi ne.yml ├── lint.yml ├── test.yml └── deploy.yml
61. 08-extract-to-template-yaml 08. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├── common.yml ├── de fi ne.yml ├── lint.yml ├── test.yml └── deploy.yml
62. 08-extract-to-template-yaml 08. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├── common.yml ├── de fi ne.yml ├── lint.yml ├── test.yml └── deploy.yml
63. 08-extract-to-template-yaml 08. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├── common.yml ├── de fi ne.yml ├── lint.yml ├── test.yml └── deploy.yml
64. 08-extract-to-template-yaml 08. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├── common.yml ├── de fi ne.yml ├── lint.yml ├── test.yml └── deploy.yml
65. 08-extract-to-template-yaml 08. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├── common.yml ├── de fi ne.yml ├── lint.yml ├── test.yml └── deploy.yml
66. 08. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├── common.yml ├── de fi ne.yml ├── lint.yml ├── test.yml └── deploy.yml 08-extract-to-template-yaml
67. include: - local: 同專案下的檔案 - remote: URL 連結 - template: GitLab 官⽅提供的模板 - fi le: 來⾃其他專案的檔案
68. include: - local: 同專案下的檔案 - remote: URL 連結 - template: GitLab 官⽅提供的模板 - fi le: 來⾃其他專案的檔案 GitLab 13.11.
69. 09. 建立共⽤流 程供其他專案 使⽤ 09-used-template-from-other-project
70. include: fi le introduced in GitLab 11.7 include multi fi le from a project in GitLab 13.6 09. 建立共⽤流 程供其他專案 使⽤ 09-used-template-from-other-project
71. 變更及覆寫參數 09. 建立共⽤流 程供其他專案 使⽤ 09-used-template-from-other-project
72. 變更及覆寫參數 09-1. 變數變 更及覆寫 09-1-overwrite-job-and-variable
73. 09-2. 從模板選 擇需要的⼯作
74. 09-2. 從模板選 擇需要的⼯作
75. Recap • 利⽤ default stage 整理所有 job 都共同的 • 利⽤ .hidden 建立模板⼯作，並⽤ extends 繼承使⽤ • 利⽤ Anchors、Reference 整理可重複使⽤的 Script • 萃取出變數 • 利⽤ Parallel Matrix 功能 • 建立多專案共⽤流程
76. Part I：GitLab CI 進階語法 Part II：加速 GitLab CI Pipeline
77. 流⽔線加速三要素
78. 流⽔線加速三要素 • ⼯作 少做點
79. 流⽔線加速三要素 • ⼯作 少做點 • 事情 作快點
80. 流⽔線加速三要素 • ⼯作 少做點 • 事情 作快點 • 作事的 多⼀點
81. 流⽔線 Pipeline 啟動到結束 哪些地⽅花了時間？
82. .gitlab-ci.yml
83. .gitlab-ci.yml
84. .gitlab-ci.yml
85. .gitlab-ci.yml Job Job Job Job Job Job Job Queue
86. .gitlab-ci.yml Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner
87. .gitlab-ci.yml Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________
88. .gitlab-ci.yml Job Job Job Job Job Job Job Queue
89. •什麼⼯作先作比較好？ .gitlab-ci.yml Job Job Job Job Job Job Job Queue
90. 什麼⼯作先做比較好？
91. 什麼⼯作先做比較好？ • 跑的快的⼯作
92. 什麼⼯作先做比較好？ • 跑的快的⼯作 • 早點做完早點確認結果
93. 什麼⼯作先做比較好？ • 跑的快的⼯作 • 早點做完早點確認結果 • 有決定性的⼯作
94. 什麼⼯作先做比較好？ • 跑的快的⼯作 • 早點做完早點確認結果 • 有決定性的⼯作 • 沒通過直接 Reject
95. •什麼⼯作先做比較好？ .gitlab-ci.yml Job Job Job Job Job Job Job Queue
96. •什麼⼯作先做比較好？ •每個⼯作都是該做的嗎？ .gitlab-ci.yml Job Job Job Job Job Job Job Queue
97. 每個⼯作都是該做的嗎？
98. 每個⼯作都是該做的嗎？ • 同⼀專案中包含前端及後端的測試
99. 每個⼯作都是該做的嗎？ • 同⼀專案中包含前端及後端的測試 • 可以考慮對應原始碼變更才執⾏對應的測試
100. 每個⼯作都是該做的嗎？ • 同⼀專案中包含前端及後端的測試 • 可以考慮對應原始碼變更才執⾏對應的測試
101. •什麼⼯作先做比較好？ •每個⼯作都是該做的嗎？ .gitlab-ci.yml Job Job Job Job Job Job Job Queue
102. •什麼⼯作先做比較好？ •每個⼯作都是該做的嗎？ •有沒有⼯作花太久時間等 待進 Queue ? .gitlab-ci.yml Job Job Job Job Job Job Job Queue
103. ⼯作花太久時間等待進 Queue? 分⽀：10-stage-without-dag-needs
104. ⼯作花太久時間等待進 Queue? build test deploy build_a test_a deploy_a build_b test_b deploy_b build_c test_c deploy_c 分⽀：10-stage-without-dag-needs
105. ⼯作花太久時間等待進 Queue? build test deploy build_a test_a deploy_a build_b test_b deploy_b build_c test_c deploy_c build_b 分⽀：10-stage-without-dag-needs
106. ⼯作花太久時間等待進 Queue? build test deploy build_a test_a deploy_a build_b test_b deploy_b build_c test_c deploy_c build_b 分⽀：10-stage-without-dag-needs
107. ⼯作花太久時間等待進 Queue? • 套⽤ DAG 機制 (GitLab 12.2） build test deploy build_a test_a deploy_a build_b test_b deploy_b build_c test_c deploy_c build_b 分⽀：10-1-stage-with-needs
108. ⼯作花太久時間等待進 Queue? • 套⽤ DAG 機制 (GitLab 12.2） build test deploy build_a test_a deploy_a build_b test_b deploy_b build_c test_c deploy_c build_b 分⽀：10-1-stage-with-needs
109. ⼯作花太久時間等待進 Queue? • 套⽤ DAG 機制 (GitLab 12.2） build test deploy build_a test_a deploy_a build_b test_b deploy_b build_c test_c deploy_c build_b 分⽀：10-1-stage-with-needs
110. .gitlab-ci.yml Job Job Job Job Job Job Job Queue
111. .gitlab-ci.yml Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner
112. Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner
113. Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner •Runner 數量夠嗎？
114. Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner •Runner 數量夠嗎？ •能執⾏的 Runner 夠嗎？
115. Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner
116. Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________
117. Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________
118. Runner Server Runner Server Runners Runner Runner Runner Runner Run Run ____________________________
119. Runner Server Runner Server Runners Runner Runner Runner Runner Run Run ____________________________ * prepare p r e pa r e p r e
120. Runner Server Runner Server Runners Runner Runner Runner Runner Run Run ____________________________ * prepare p r e pa r e p r e * git g i t gi t r
121. Runner Server Runner Server Runners Runner Runner Runner Runner Run Run ____________________________ * prepare p r e pa r e p r e * git g i t gi t r * restore re s t o
122. Runner Server Runner Server Runners Runner Runner Runner Runner Run Run ____________________________ * prepare p r e pa r e p r e * git g i t gi t r * restore re s t o * download do wnlo ad
123. Runner Server Runner Server Runners Runner Runner Runner Runner Run Run ____________________________ * prepare p r e pa r e p r e * git g i t gi t r * restore re s t o * download do wnlo ad * script s c r i p t
124. Runner Server Runner Server Runners Runner Runner Runner Runner Run Run ____________________________ * prepare p r e pa r e p r e * git g i t gi t r * restore re s t o * download do wnlo ad * script s c r i p t * package p ack a ge
125. Runner Server Runner Server Runners Runner Runner Runner Runner Run Run ____________________________ * prepare p r e pa r e p r e * git g i t gi t r * restore re s t o * download do wnlo ad * script s c r i p t * package p ack a ge * saving s a vin
126. Runner Server Runner Server Runners Runner Runner Runner Runner Run Run ____________________________ * prepare p r e pa r e p r e * git g i t gi t r * restore re s t o * download do wnlo ad * script s c r i p t * package p ack a ge * saving s a vin * uploading u p load i ng
127. Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var
128. •Network Performance ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var
129. •Network Performance ? •Docker image size ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var
130. Docker Image Size ? 分⽀：11-compare-difference-image
131. •Network Performance ? •Docker image size ? •Git Repo Size ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var
132. •Network Performance ? •Docker image size ? •Git Repo Size ? •Programming Language Dependence Package Size ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var
133. •Network Performance ? •Docker image size ? •Git Repo Size ? •Programming Language Dependence Package Size ? •Artifacts Size ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var
134. •Network Performance ? •Docker image size ? •Git Repo Size ? •Programming Language Dependence Package Size ? •Artifacts Size ? •Script Performance ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var
135. Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var
136. .gitlab-ci.yml Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var
137. Q&A