Advanced GitLab Workshop - 談 pipeline 調教與進階 GitLab CI 語法

Slide 1

Slide 1 text

Advanced GitLab CI 談 pipeline 調教與進階 GitLab CI 墨嗓 (陳佑⽵) 2021/11/24 @ DevOpsDays Taipei 2021

Slide 2

Slide 2 text

About Me • 陳佑⽵，朋友都叫我墨嗓(Mouson)。專注於資訊科技整合、應⽤、開發⽅法及研究的T型⼈。妞妞

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

DevOpsDaysGitLab

Slide 5

Slide 5 text

本⽇練習範例 https://bit.ly/DevOpsDaysGitLab https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24

Slide 6

Slide 6 text

Fork https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24 https://bit.ly/DevOpsDaysGitLab

Slide 7

Slide 7 text

到⾃⼰的空間 https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24 https://bit.ly/DevOpsDaysGitLab

Slide 8

Slide 8 text

皆已建立分⽀ https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24 https://bit.ly/DevOpsDaysGitLab

Slide 9

Slide 9 text

進入 Pipeline https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24 https://bit.ly/DevOpsDaysGitLab

Slide 10

Slide 10 text

Run Pipeline https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24 https://bit.ly/DevOpsDaysGitLab

Slide 11

Slide 11 text

Run Pipeline https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24 https://bit.ly/DevOpsDaysGitLab

Slide 12

Slide 12 text

使⽤ Pipeline Editor https://gitlab.com/mouson-gitlab-playground/gitlab-ci-workshop-2021-11-24 https://bit.ly/DevOpsDaysGitLab

Slide 13

Slide 13 text

Part I：GitLab CI 進階語法 Part II：加速 GitLab CI Pipeline

Slide 14

Slide 14 text

Part I：GitLab CI 進階語法 Part II：加速 GitLab CI Pipeline

Slide 15

Slide 15 text

從前從前.... 使⽤ GitLab CI ⾃動檢查格式

Slide 16

Slide 16 text

加入⾃動測試

Slide 17

Slide 17 text

加入⾃動部署

Slide 18

Slide 18 text

再增加⼀種版本... 分⽀：00-init

Slide 19

Slide 19 text

越來越難以維護的 .gitlab-ci.yml 該怎麼整理？

Slide 20

Slide 20 text

怎麼整理？針對 v73, v74 分別有： lint、test、deploy 三種類型的⼯作

Slide 21

Slide 21 text

01. 擷取⼀致的部分到 Default: 擷取⼀致的 script 變為 default: 分⽀：01-default

Slide 22

Slide 22 text

01. 擷取⼀致的部分到 Default: 擷取⼀致的 script 變為 default: 分⽀：01-default

Slide 23

Slide 23 text

Default: 1. Pipeline 中所有⼯作都會套⽤ 2. 如不符使⽤，可以覆寫 3. ⽀援：image、services、 before_script、after_script、tags、 cache、artifacts、retry、timeout、 interruptible 分⽀：01-default

Slide 24

Slide 24 text

Default: 不想沿⽤ Default，可以使⽤ inherit:default https://docs.gitlab.com/ee/ci/yaml/index.html#inheritdefault

Slide 25

Slide 25 text

02. 整理 script 共同之程序整理出來的 Script，可以做程序的分組，⽅便後續使⽤分⽀：02-extract-to-before-script

Slide 26

Slide 26 text

2. 整理 script 共同之程序整理出來的 Script，可以做程序的分組，⽅便後續使⽤分⽀：02-extract-to-before-script

Slide 27

Slide 27 text

02. 整理 script 共同之程序整理出來的 Script，可以做程序的分組，⽅便後續使⽤分⽀：02-extract-to-before-script

Slide 28

Slide 28 text

03. 擷取重複段落以繼承再利⽤利⽤ hidden job 整理 job 共同的部分並使⽤ extends 繼承分⽀：03-extract-before-script-t...

Slide 29

Slide 29 text

03. 擷取重複段落以繼承再利⽤利⽤ hidden job 整理 job 共同的部分並使⽤ extends 繼承分⽀：03-extract-before-script-t...

Slide 30

Slide 30 text

03. 擷取重複段落以繼承再利⽤利⽤ hidden job 整理 job 共同的部分並使⽤ extends 繼承分⽀：03-extract-before-script-t...

Slide 31

Slide 31 text

hidden job 1. 以「.」開頭的⼯作名稱：整個⼯作都不執⾏分⽀：03-extract-before-script-t...

Slide 32

Slide 32 text

extends: 分⽀：03-extract-before-script-t...

Slide 33

Slide 33 text

extends: 1. 把繼承來源的內容抄⼀次分⽀：03-extract-before-script-t...

Slide 34

Slide 34 text

extends: 1. 把繼承來源的內容抄⼀次 2. 可以允許既成多個來源分⽀：03-extract-before-script-t...

Slide 35

Slide 35 text

extends: 1. 把繼承來源的內容抄⼀次 2. 可以允許既成多個來源 3. 重複的內容：分⽀：03-extract-before-script-t...

Slide 36

Slide 36 text

extends: 1. 把繼承來源的內容抄⼀次 2. 可以允許既成多個來源 3. 重複的內容： script: 後蓋前分⽀：03-extract-before-script-t...

Slide 37

Slide 37 text

extends: 1. 把繼承來源的內容抄⼀次 2. 可以允許既成多個來源 3. 重複的內容： script: 後蓋前 variables: 取合集後蓋前分⽀：03-extract-before-script-t...

Slide 38

Slide 38 text

04.利⽤Anchors 整理可複⽤的 script 分⽀：04-extract-prepare-process

Slide 39

Slide 39 text

04.利⽤Anchors 整理可複⽤的 script 分⽀：04-extract-prepare-process

Slide 40

Slide 40 text

04.利⽤Anchors 整理可複⽤的 script 分⽀：04-extract-prepare-process

Slide 41

Slide 41 text

Anchors 1. 可允許多⾏ 2. 可在 script 中使⽤分⽀：04-extract-prepare-process

Slide 42

Slide 42 text

Anchors 1. 可允許多⾏ 2. 可在 script 中使⽤分⽀：04-extract-prepare-process

Slide 43

Slide 43 text

Anchors 1. 可允許多⾏ 2. 可在 script 中使⽤分⽀：04-extract-prepare-process

Slide 44

Slide 44 text

04-02. 利⽤ !reference 整理可複⽤的 script (GitLab 13.9) 04-02-extract-prepare-process https://docs.gitlab.com/ee/ci/yaml/yaml_optimization.html

Slide 45

Slide 45 text

!reference vs. anchor tags !reference 可以參考 include 的 yaml 04-02-extract-prepare-process https://docs.gitlab.com/ee/ci/yaml/yaml_optimization.html

Slide 46

Slide 46 text

05-1. 擷取為變數控制 05-1-extract-to-variables

Slide 47

Slide 47 text

05-1. 擷取為變數控制 05-1-extract-to-variables

Slide 48

Slide 48 text

05-1. 擷取為變數控制 05-1-extract-to-variables

Slide 49

Slide 49 text

05-2. 建立同類⼯作模板 tmpl extends: 可以允許多層次繼承 05-2-extract-job-to-template

Slide 50

Slide 50 text

05-2. 建立同類⼯作模板 tmpl extends: 可以允許多層次繼承 05-2-extract-job-to-template

Slide 51

Slide 51 text

extends: 可以允許多層次繼承 06. 使⽤ Parallel Matrix 06-extract-to-parallel-matrix

Slide 52

Slide 52 text

06. 使⽤ Parallel Matrix 06-extract-to-parallel-matrix

Slide 53

Slide 53 text

06. 使⽤ Parallel Matrix 06-extract-to-parallel-matrix

Slide 54

Slide 54 text

07. 擷取 Parallel Matrix 為模板供繼承 07-extract-version-de fi ne-to-....

Slide 55

Slide 55 text

07. 擷取 Parallel Matrix 為模板供繼承 07-extract-version-de fi ne-to-....

Slide 56

Slide 56 text

07. 擷取 Parallel Matrix 為模板供繼承 entends: 允許既成多個來源 07-extract-version-de fi ne-to-....

Slide 57

Slide 57 text

08. 拆分檔案利⽤ include 載入 08-extract-to-template-yaml

Slide 58

Slide 58 text

08. 拆分檔案利⽤ include 載入 08-extract-to-template-yaml

Slide 59

Slide 59 text

08. 拆分檔案利⽤ include 載入 ├── .gitlab-ci.yml └── templates ├── common.yml ├── de fi ne.yml ├── lint.yml ├── test.yml └── deploy.yml 08-extract-to-template-yaml

Slide 60

Slide 60 text

08-extract-to-template-yaml 08. 拆分檔案利⽤ include 載入 ├── .gitlab-ci.yml └── templates ├── common.yml ├── de fi ne.yml ├── lint.yml ├── test.yml └── deploy.yml

Slide 61

Slide 61 text

Slide 62

Slide 62 text

Slide 63

Slide 63 text

Slide 64

Slide 64 text

Slide 65

Slide 65 text

Slide 66

Slide 66 text

Slide 67

Slide 67 text

include: - local: 同專案下的檔案 - remote: URL 連結 - template: GitLab 官⽅提供的模板 - fi le: 來⾃其他專案的檔案

Slide 68

Slide 68 text

include: - local: 同專案下的檔案 - remote: URL 連結 - template: GitLab 官⽅提供的模板 - fi le: 來⾃其他專案的檔案 GitLab 13.11.

Slide 69

Slide 69 text

09. 建立共⽤流程供其他專案使⽤ 09-used-template-from-other-project

Slide 70

Slide 70 text

include: fi le introduced in GitLab 11.7 include multi fi le from a project in GitLab 13.6 09. 建立共⽤流程供其他專案使⽤ 09-used-template-from-other-project

Slide 71

Slide 71 text

變更及覆寫參數 09. 建立共⽤流程供其他專案使⽤ 09-used-template-from-other-project

Slide 72

Slide 72 text

變更及覆寫參數 09-1. 變數變更及覆寫 09-1-overwrite-job-and-variable

Slide 73

Slide 73 text

09-2. 從模板選擇需要的⼯作

Slide 74

Slide 74 text

09-2. 從模板選擇需要的⼯作

Slide 75

Slide 75 text

Recap • 利⽤ default stage 整理所有 job 都共同的 • 利⽤ .hidden 建立模板⼯作，並⽤ extends 繼承使⽤ • 利⽤ Anchors、Reference 整理可重複使⽤的 Script • 萃取出變數 • 利⽤ Parallel Matrix 功能 • 建立多專案共⽤流程

Slide 76

Slide 76 text

Part I：GitLab CI 進階語法 Part II：加速 GitLab CI Pipeline

Slide 77

Slide 77 text

流⽔線加速三要素

Slide 78

Slide 78 text

流⽔線加速三要素 • ⼯作少做點

Slide 79

Slide 79 text

流⽔線加速三要素 • ⼯作少做點 • 事情作快點

Slide 80

Slide 80 text

流⽔線加速三要素 • ⼯作少做點 • 事情作快點 • 作事的多⼀點

Slide 81

Slide 81 text

流⽔線 Pipeline 啟動到結束哪些地⽅花了時間？

Slide 82

Slide 82 text

.gitlab-ci.yml

Slide 83

Slide 83 text

.gitlab-ci.yml

Slide 84

Slide 84 text

.gitlab-ci.yml

Slide 85

Slide 85 text

.gitlab-ci.yml Job Job Job Job Job Job Job Queue

Slide 86

Slide 86 text

.gitlab-ci.yml Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner

Slide 87

Slide 87 text

.gitlab-ci.yml Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________

Slide 88

Slide 88 text

.gitlab-ci.yml Job Job Job Job Job Job Job Queue

Slide 89

Slide 89 text

•什麼⼯作先作比較好？ .gitlab-ci.yml Job Job Job Job Job Job Job Queue

Slide 90

Slide 90 text

什麼⼯作先做比較好？

Slide 91

Slide 91 text

什麼⼯作先做比較好？ • 跑的快的⼯作

Slide 92

Slide 92 text

什麼⼯作先做比較好？ • 跑的快的⼯作 • 早點做完早點確認結果

Slide 93

Slide 93 text

什麼⼯作先做比較好？ • 跑的快的⼯作 • 早點做完早點確認結果 • 有決定性的⼯作

Slide 94

Slide 94 text

什麼⼯作先做比較好？ • 跑的快的⼯作 • 早點做完早點確認結果 • 有決定性的⼯作 • 沒通過直接 Reject

Slide 95

Slide 95 text

•什麼⼯作先做比較好？ .gitlab-ci.yml Job Job Job Job Job Job Job Queue

Slide 96

Slide 96 text

•什麼⼯作先做比較好？ •每個⼯作都是該做的嗎？ .gitlab-ci.yml Job Job Job Job Job Job Job Queue

Slide 97

Slide 97 text

每個⼯作都是該做的嗎？

Slide 98

Slide 98 text

每個⼯作都是該做的嗎？ • 同⼀專案中包含前端及後端的測試

Slide 99

Slide 99 text

每個⼯作都是該做的嗎？ • 同⼀專案中包含前端及後端的測試 • 可以考慮對應原始碼變更才執⾏對應的測試

Slide 100

Slide 100 text

每個⼯作都是該做的嗎？ • 同⼀專案中包含前端及後端的測試 • 可以考慮對應原始碼變更才執⾏對應的測試

Slide 101

Slide 101 text

•什麼⼯作先做比較好？ •每個⼯作都是該做的嗎？ .gitlab-ci.yml Job Job Job Job Job Job Job Queue

Slide 102

Slide 102 text

•什麼⼯作先做比較好？ •每個⼯作都是該做的嗎？ •有沒有⼯作花太久時間等待進 Queue ? .gitlab-ci.yml Job Job Job Job Job Job Job Queue

Slide 103

Slide 103 text

⼯作花太久時間等待進 Queue? 分⽀：10-stage-without-dag-needs

Slide 104

Slide 104 text

⼯作花太久時間等待進 Queue? build test deploy build_a test_a deploy_a build_b test_b deploy_b build_c test_c deploy_c 分⽀：10-stage-without-dag-needs

Slide 105

Slide 105 text

⼯作花太久時間等待進 Queue? build test deploy build_a test_a deploy_a build_b test_b deploy_b build_c test_c deploy_c build_b 分⽀：10-stage-without-dag-needs

Slide 106

Slide 106 text

⼯作花太久時間等待進 Queue? build test deploy build_a test_a deploy_a build_b test_b deploy_b build_c test_c deploy_c build_b 分⽀：10-stage-without-dag-needs

Slide 107

Slide 107 text

⼯作花太久時間等待進 Queue? • 套⽤ DAG 機制 (GitLab 12.2） build test deploy build_a test_a deploy_a build_b test_b deploy_b build_c test_c deploy_c build_b 分⽀：10-1-stage-with-needs

Slide 108

Slide 108 text

Slide 109

Slide 109 text

Slide 110

Slide 110 text

.gitlab-ci.yml Job Job Job Job Job Job Job Queue

Slide 111

Slide 111 text

.gitlab-ci.yml Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner

Slide 112

Slide 112 text

Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner

Slide 113

Slide 113 text

Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner •Runner 數量夠嗎？

Slide 114

Slide 114 text

Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner •Runner 數量夠嗎？ •能執⾏的 Runner 夠嗎？

Slide 115

Slide 115 text

Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner

Slide 116

Slide 116 text

Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________

Slide 117

Slide 117 text

Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________

Slide 118

Slide 118 text

Runner Server Runner Server Runners Runner Runner Runner Runner Run Run ____________________________

Slide 119

Slide 119 text

Runner Server Runner Server Runners Runner Runner Runner Runner Run Run ____________________________ * prepare p r e pa r e p r e

Slide 120

Slide 120 text

Runner Server Runner Server Runners Runner Runner Runner Runner Run Run ____________________________ * prepare p r e pa r e p r e * git g i t gi t r

Slide 121

Slide 121 text

Runner Server Runner Server Runners Runner Runner Runner Runner Run Run ____________________________ * prepare p r e pa r e p r e * git g i t gi t r * restore re s t o

Slide 122

Slide 122 text

Runner Server Runner Server Runners Runner Runner Runner Runner Run Run ____________________________ * prepare p r e pa r e p r e * git g i t gi t r * restore re s t o * download do wnlo ad

Slide 123

Slide 123 text

Runner Server Runner Server Runners Runner Runner Runner Runner Run Run ____________________________ * prepare p r e pa r e p r e * git g i t gi t r * restore re s t o * download do wnlo ad * script s c r i p t

Slide 124

Slide 124 text

Slide 125

Slide 125 text

Slide 126

Slide 126 text

Slide 127

Slide 127 text

Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var

Slide 128

Slide 128 text

•Network Performance ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var

Slide 129

Slide 129 text

•Network Performance ? •Docker image size ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var

Slide 130

Slide 130 text

Docker Image Size ? 分⽀：11-compare-difference-image

Slide 131

Slide 131 text

•Network Performance ? •Docker image size ? •Git Repo Size ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var

Slide 132

Slide 132 text

•Network Performance ? •Docker image size ? •Git Repo Size ? •Programming Language Dependence Package Size ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var

Slide 133

Slide 133 text

•Network Performance ? •Docker image size ? •Git Repo Size ? •Programming Language Dependence Package Size ? •Artifacts Size ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var

Slide 134

Slide 134 text

•Network Performance ? •Docker image size ? •Git Repo Size ? •Programming Language Dependence Package Size ? •Artifacts Size ? •Script Performance ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var

Slide 135

Slide 135 text

Slide 136

Slide 136 text

.gitlab-ci.yml Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up fi le base var

Slide 137

Slide 137 text

Q&A