Tweet
Tweet

Slide 1

Slide 1 text

深入 GitLab CI 談描述檔重構與流⽔線加速 墨嗓 (陳佑⽵) 2020/11/28 @ 台中⾃由軟體愛好者社群

Slide 2

Slide 2 text

About Me • 陳佑⽵，朋友都叫我墨嗓(Mouson)。 專注於資訊科技整合、應⽤、開發⽅ 法及研究的T型⼈。 妞妞

Slide 3

Slide 3 text

歡迎加入 GitLab Hero

Slide 4

Slide 4 text

. 社群 . https://fb.com/groups/GitLabTaipei https://t.me/GitLabTaiwan

Slide 5

Slide 5 text

進入主題

Slide 6

Slide 6 text

深入 GitLab CI • Part I: 流⽔線 (Pipeline) 加速 • Part II: .gitlab-ci.yml 描述檔如何重構

Slide 7

Slide 7 text

怎麼讓流⽔線 (Pipeline) 更快⼀點？ Part I

Slide 8

Slide 8 text

流⽔線加速三要素

Slide 9

Slide 9 text

流⽔線加速三要素 • ⼯作 少做點

Slide 10

Slide 10 text

流⽔線加速三要素 • ⼯作 少做點 • 事情 作快點

Slide 11

Slide 11 text

流⽔線加速三要素 • ⼯作 少做點 • 事情 作快點 • 作事的 多⼀點

Slide 12

Slide 12 text

流⽔線 Pipeline 啟動到結束 哪些地⽅花了時間？

Slide 13

Slide 13 text

拆解

Slide 14

Slide 14 text

.gitlab-ci.yml

Slide 15

Slide 15 text

.gitlab-ci.yml

Slide 16

Slide 16 text

.gitlab-ci.yml

Slide 17

Slide 17 text

.gitlab-ci.yml Job Job Job Job Job Job Job Queue

Slide 18

Slide 18 text

.gitlab-ci.yml Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner

Slide 19

Slide 19 text

.gitlab-ci.yml Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________

Slide 20

Slide 20 text

.gitlab-ci.yml Job Job Job Job Job Job Job Queue

Slide 21

Slide 21 text

•什麼⼯作先作比較好？ .gitlab-ci.yml Job Job Job Job Job Job Job Queue

Slide 22

Slide 22 text

什麼⼯作先做比較好？

Slide 23

Slide 23 text

什麼⼯作先做比較好？ • 跑的快的⼯作

Slide 24

Slide 24 text

什麼⼯作先做比較好？ • 跑的快的⼯作 • 早點做完早點確認結果

Slide 25

Slide 25 text

什麼⼯作先做比較好？ • 跑的快的⼯作 • 早點做完早點確認結果 • 有決定性的⼯作

Slide 26

Slide 26 text

什麼⼯作先做比較好？ • 跑的快的⼯作 • 早點做完早點確認結果 • 有決定性的⼯作 • 沒通過直接 Reject

Slide 27

Slide 27 text

•什麼⼯作先做比較好？ .gitlab-ci.yml Job Job Job Job Job Job Job Queue

Slide 28

Slide 28 text

•什麼⼯作先做比較好？ •每個⼯作都是該做的嗎？ .gitlab-ci.yml Job Job Job Job Job Job Job Queue

Slide 29

Slide 29 text

每個⼯作都是該做的嗎？

Slide 30

Slide 30 text

每個⼯作都是該做的嗎？ • 同⼀專案中包含前端及後端的測試

Slide 31

Slide 31 text

每個⼯作都是該做的嗎？ • 同⼀專案中包含前端及後端的測試 • 可以考慮對應原始碼變更才執⾏對應的測試

Slide 32

Slide 32 text

每個⼯作都是該做的嗎？ • 同⼀專案中包含前端及後端的測試 • 可以考慮對應原始碼變更才執⾏對應的測試

Slide 33

Slide 33 text

•什麼⼯作先做比較好？ •每個⼯作都是該做的嗎？ .gitlab-ci.yml Job Job Job Job Job Job Job Queue

Slide 34

Slide 34 text

•什麼⼯作先做比較好？ •每個⼯作都是該做的嗎？ •有沒有⼯作花太久時間等 待進 Queue ? .gitlab-ci.yml Job Job Job Job Job Job Job Queue

Slide 35

Slide 35 text

⼯作花太久時間等待進 Queue?

Slide 36

Slide 36 text

⼯作花太久時間等待進 Queue? build test deploy build_a test_a deploy_a build_b test_b deploy_b build_c test_c deploy_c

Slide 37

Slide 37 text

⼯作花太久時間等待進 Queue? build test deploy build_a test_a deploy_a build_b test_b deploy_b build_c test_c deploy_c build_b

Slide 38

Slide 38 text

⼯作花太久時間等待進 Queue? build test deploy build_a test_a deploy_a build_b test_b deploy_b build_c test_c deploy_c build_b

Slide 39

Slide 39 text

⼯作花太久時間等待進 Queue? • 套⽤ DAG 機制 (GitLab 12.2） build test deploy build_a test_a deploy_a build_b test_b deploy_b build_c test_c deploy_c build_b

Slide 40

Slide 40 text

⼯作花太久時間等待進 Queue? • 套⽤ DAG 機制 (GitLab 12.2） build test deploy build_a test_a deploy_a build_b test_b deploy_b build_c test_c deploy_c build_b

Slide 41

Slide 41 text

⼯作花太久時間等待進 Queue? • 套⽤ DAG 機制 (GitLab 12.2） build test deploy build_a test_a deploy_a build_b test_b deploy_b build_c test_c deploy_c build_b

Slide 42

Slide 42 text

.gitlab-ci.yml Job Job Job Job Job Job Job Queue

Slide 43

Slide 43 text

.gitlab-ci.yml Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner

Slide 44

Slide 44 text

Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner

Slide 45

Slide 45 text

Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner •Runner 數量夠嗎？

Slide 46

Slide 46 text

Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner •Runner 數量夠嗎？ •能執⾏的 Runner 夠嗎？

Slide 47

Slide 47 text

Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner

Slide 48

Slide 48 text

Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________

Slide 49

Slide 49 text

Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________

Slide 50

Slide 50 text

Runner Server Runner Server Runners Runner Runner Runner Runner Run R u n ____________________________

Slide 51

Slide 51 text

Runner Server Runner Server Runners Runner Runner Runner Runner Run R u n ____________________________ * prepare p r e p a r e p r e

Slide 52

Slide 52 text

Runner Server Runner Server Runners Runner Runner Runner Runner Run R u n ____________________________ * prepare p r e p a r e p r e * git g i t g i t r

Slide 53

Slide 53 text

Runner Server Runner Server Runners Runner Runner Runner Runner Run R u n ____________________________ * prepare p r e p a r e p r e * git g i t g i t r * restore r e s t o

Slide 54

Slide 54 text

Runner Server Runner Server Runners Runner Runner Runner Runner Run R u n ____________________________ * prepare p r e p a r e p r e * git g i t g i t r * restore r e s t o * download d o w n l o a d

Slide 55

Slide 55 text

Runner Server Runner Server Runners Runner Runner Runner Runner Run R u n ____________________________ * prepare p r e p a r e p r e * git g i t g i t r * restore r e s t o * download d o w n l o a d * script s c r i p t

Slide 56

Slide 56 text

Runner Server Runner Server Runners Runner Runner Runner Runner Run R u n ____________________________ * prepare p r e p a r e p r e * git g i t g i t r * restore r e s t o * download d o w n l o a d * script s c r i p t * package p a c k a g e

Slide 57

Slide 57 text

Runner Server Runner Server Runners Runner Runner Runner Runner Run R u n ____________________________ * prepare p r e p a r e p r e * git g i t g i t r * restore r e s t o * download d o w n l o a d * script s c r i p t * package p a c k a g e * saving s a v i n

Slide 58

Slide 58 text

Runner Server Runner Server Runners Runner Runner Runner Runner Run R u n ____________________________ * prepare p r e p a r e p r e * git g i t g i t r * restore r e s t o * download d o w n l o a d * script s c r i p t * package p a c k a g e * saving s a v i n * uploading u p l o a d i n g

Slide 59

Slide 59 text

Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up f i le base var

Slide 60

Slide 60 text

•Network Performance ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execute * package install * saving cache * uploading artifacts * cleaning up f i le base var

Slide 61

Slide 61 text

•Network Performance ? •Docker image size ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execut e * package install * saving cache * uploading artifacts * cleaning up f i le base var

Slide 62

Slide 62 text

Docker Image Size ?

Slide 63

Slide 63 text

Docker Image Size ?

Slide 64

Slide 64 text

•Network Performance ? •Docker image size ? •Git Repo Size ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execut e * package install * saving cache * uploading artifacts * cleaning up f i le base var

Slide 65

Slide 65 text

•Network Performance ? •Docker image size ? •Git Repo Size ? •Programming Language Dependence Package Size ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execut e * package install * saving cache * uploading artifacts * cleaning up f i le base var

Slide 66

Slide 66 text

•Network Performance ? •Docker image size ? •Git Repo Size ? •Programming Language Dependence Package Size ? •Artifacts Size ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execut e * package install * saving cache * uploading artifacts * cleaning up f i le base var

Slide 67

Slide 67 text

•Network Performance ? •Docker image size ? •Git Repo Size ? •Programming Language Dependence Package Size ? •Artifacts Size ? •Script Performance ? Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execut e * package install * saving cache * uploading artifacts * cleaning up f i le base var

Slide 68

Slide 68 text

Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execut e * package install * saving cache * uploading artifacts * cleaning up f i le base var

Slide 69

Slide 69 text

.gitlab-ci.yml Job Job Job Job Job Job Job Queue Runner Server Runner Server Runners Runner Runner Runner Runner Run Job ____________________________ * prepare execute env * git repo fetch * restore cache * download artifacts * script execut e * package install * saving cache * uploading artifacts * cleaning up f i le base var

Slide 70

Slide 70 text

Part I Q&A

Slide 71

Slide 71 text

.gitlab-ci.yml 描述檔 如何重構？ Part II

Slide 72

Slide 72 text

從前從前.... 使⽤ GitLab CI ⾃動檢查格式

Slide 73

Slide 73 text

加入⾃動測試

Slide 74

Slide 74 text

加入⾃動部署

Slide 75

Slide 75 text

再增加 ⼀種版本...

Slide 76

Slide 76 text

越來越難以維護的 .gitlab-ci.yml 該怎麼整理？

Slide 77

Slide 77 text

怎麼整理？ 針對 v73, v74 分別有： lint、test、deploy 三種類型的⼯作

Slide 78

Slide 78 text

1. 擷取⼀致的 部分到 Default: 擷取⼀致的 script 變為 default:

Slide 79

Slide 79 text

1. 擷取⼀致的 部分到 Default: 擷取⼀致的 script 變為 default:

Slide 80

Slide 80 text

Default: 1. Pipeline 中所有⼯作都會套⽤ 2. 如不符使⽤，可以覆寫 3. ⽀援：image、services、 before_script、after_script、tags、 cache、artifacts、retry、timeout、 interruptible

Slide 81

Slide 81 text

2. 整理 script 共同之程序 整理出來的 Script，可以做程序的分 組，⽅便後續使⽤

Slide 82

Slide 82 text

2. 整理 script 共同之程序 整理出來的 Script，可以做程序的分 組，⽅便後續使⽤

Slide 83

Slide 83 text

2. 整理 script 共同之程序 整理出來的 Script，可以做程序的分 組，⽅便後續使⽤

Slide 84

Slide 84 text

3. 擷取重複段 落以繼承再利⽤ 利⽤ hidden job 整理 job 共同的部分 並使⽤ extends 繼承

Slide 85

Slide 85 text

3. 擷取重複段 落以繼承再利⽤ 利⽤ hidden job 整理 job 共同的部分 並使⽤ extends 繼承

Slide 86

Slide 86 text

3. 擷取重複段 落以繼承再利⽤ 利⽤ hidden job 整理 job 共同的部分 並使⽤ extends 繼承

Slide 87

Slide 87 text

hidden job 1. 以 「.」開頭的⼯作名稱： 整個⼯作都不執⾏

Slide 88

Slide 88 text

extends:

Slide 89

Slide 89 text

extends: 1. 把繼承來源的內容抄⼀次

Slide 90

Slide 90 text

extends: 1. 把繼承來源的內容抄⼀次 2. 可以允許既成多個來源

Slide 91

Slide 91 text

extends: 1. 把繼承來源的內容抄⼀次 2. 可以允許既成多個來源 3. 重複的內容：

Slide 92

Slide 92 text

extends: 1. 把繼承來源的內容抄⼀次 2. 可以允許既成多個來源 3. 重複的內容： script: 後蓋前

Slide 93

Slide 93 text

extends: 1. 把繼承來源的內容抄⼀次 2. 可以允許既成多個來源 3. 重複的內容： script: 後蓋前 variables: 取合集後蓋前

Slide 94

Slide 94 text

4.利⽤Anchors 整理可複⽤的 script

Slide 95

Slide 95 text

4.利⽤Anchors 整理可複⽤的 script

Slide 96

Slide 96 text

4.利⽤Anchors 整理可複⽤的 script

Slide 97

Slide 97 text

Anchors 1. 可允許多⾏ 2. 可在 script 中使⽤

Slide 98

Slide 98 text

Anchors 1. 可允許多⾏ 2. 可在 script 中使⽤

Slide 99

Slide 99 text

Anchors 1. 可允許多⾏ 2. 可在 script 中使⽤

Slide 100

Slide 100 text

5. 建立同類⼯ 作模板 tmpl

Slide 101

Slide 101 text

5. 建立同類⼯ 作模板 tmpl

Slide 102

Slide 102 text

5. 建立同類⼯ 作模板 tmpl extends: 可以允許多層次繼承

Slide 103

Slide 103 text

6. 擷取為 變數控制

Slide 104

Slide 104 text

6. 擷取為 變數控制

Slide 105

Slide 105 text

6. 擷取為 變數控制

Slide 106

Slide 106 text

6. 擷取為 變數控制

Slide 107

Slide 107 text

7. 整理⾄ Template 中

Slide 108

Slide 108 text

7. 整理⾄ Template 中

Slide 109

Slide 109 text

7. 整理⾄ Template 中

Slide 110

Slide 110 text

8. 使⽤ Parallel Matrix

Slide 111

Slide 111 text

8. 使⽤ Parallel Matrix

Slide 112

Slide 112 text

8. 使⽤ Parallel Matrix

Slide 113

Slide 113 text

8. 使⽤ Parallel Matrix

Slide 114

Slide 114 text

9. 擷取 Parallel Matrix 為模板供 繼承

Slide 115

Slide 115 text

9. 擷取 Parallel Matrix 為模板供 繼承

Slide 116

Slide 116 text

9. 擷取 Parallel Matrix 為模板供 繼承 entends: 允許既成多個來源

Slide 117

Slide 117 text

10. 拆分檔案利 ⽤ include 載入

Slide 118

Slide 118 text

10. 拆分檔案利 ⽤ include 載入

Slide 119

Slide 119 text

10. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├── common.yml ├── de f i ne.yml ├── lint.yml ├── test.yml └── deploy.yml

Slide 120

Slide 120 text

10. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├── common.yml ├── de f i ne.yml ├── lint.yml ├── test.yml └── deploy.yml

Slide 121

Slide 121 text

10. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├── common.yml ├── de f i ne.yml ├── lint.yml ├── test.yml └── deploy.yml

Slide 122

Slide 122 text

10. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├── common.yml ├── de f i ne.yml ├── lint.yml ├── test.yml └── deploy.yml

Slide 123

Slide 123 text

10. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├── common.yml ├── de f i ne.yml ├── lint.yml ├── test.yml └── deploy.yml

Slide 124

Slide 124 text

10. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├── common.yml ├── de f i ne.yml ├── lint.yml ├── test.yml └── deploy.yml

Slide 125

Slide 125 text

10. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├── common.yml ├── de f i ne.yml ├── lint.yml ├── test.yml └── deploy.yml

Slide 126

Slide 126 text

10. 拆分檔案利 ⽤ include 載入 ├── .gitlab-ci.yml └── templates ├── common.yml ├── de f i ne.yml ├── lint.yml ├── test.yml └── deploy.yml

Slide 127

Slide 127 text

include: - local: 同專案下的檔案 - remote: URL 連結 - template: GitLab 官⽅提供的模板 - f i le: 來⾃其他專案的檔案

Slide 128

Slide 128 text

11. 建立共⽤流 程供其他專案 使⽤

Slide 129

Slide 129 text

include: f i le introduced in GitLab 11.7 include multi f i le from a project in GitLab 13.6 11. 建立共⽤流 程供其他專案 使⽤

Slide 130

Slide 130 text

變更及覆寫參數 11. 建立共⽤流 程供其他專案 使⽤

Slide 131

Slide 131 text

變更及覆寫參數 11. 建立共⽤流 程供其他專案 使⽤

Slide 132

Slide 132 text

12. 從模板選 擇需要的⼯作

Slide 133

Slide 133 text

12. 從模板選 擇需要的⼯作

Slide 134

Slide 134 text

Recap • 利⽤ default stage 整理所有 job 都共同的 • 利⽤ .hidden 建立模板⼯作，並⽤ extends 繼承使⽤ • 利⽤ Anchors 整理可重複使⽤的 Script • 萃取出變數 • 利⽤ Parallel Matrix 功能 • 建立多專案共⽤流程 • Sample Source Code: https://bit.ly/3mTiBgJ

Slide 135

Slide 135 text

Sample Code • https://bit.ly/3mTiBgJ

Slide 136

Slide 136 text

歡迎加入 GitLab Hero

Slide 137

Slide 137 text

Q&A