Tweet
Tweet

Slide 1

Slide 1 text

如何設計⼀套具備 Container 容器化技術的 CI/CD 平台 Bo-Yi Wu @ Mediatek https://blog.wu-boy.com/ 2024/07/03

Slide 2

Slide 2 text

(JU)VC"DUJPOT +FOLJOT (JU-BC$*

Slide 3

Slide 3 text

What is Drone?

Slide 4

Slide 4 text

Drone 特性 • 使⽤ Go 語⾔編寫，所有組件都運⾏在 Docker 容器中 • ⽀持多種操作系統和架構（Linux、Windows、Mac） • 可擴展架構，具⾃動縮放功能，根據需求管理減少雲成本 • 可選擇安裝在本地或作為具有附加功能的 SaaS 產品使⽤ • 可使⽤任何語⾔編寫插件 • 管道使⽤簡單語法的 YAML 格式定義。 • 開源

Slide 5

Slide 5 text

Architecture

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Drone Runner

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

4UBSUUIF1PMMFS"HFOU

Slide 11

Slide 11 text

poller := &poller.Poller{ Client: cli, Dispatch: runner.Run, Filter: &client.Filter{ Kind: resource.Kind, Type: resource.Type, OS: config.Platform.OS, Arch: config.Platform.Arch, Variant: config.Platform.Variant, Kernel: config.Platform.Kernel, Labels: config.Runner.Labels, }, } ࠜᎦෆಉܥ౷㐫ଶ ܾఆࣥߦಛఆ೚຿ ՄҎઃఆ(16 $16ෆಉओػ

Slide 12

Slide 12 text

func (p *Poller) Poll(ctx context.Context, n int) { var wg sync.WaitGroup for i := 0; i < n; i++ { wg.Add(1) go func(i int) { for { select { case <-ctx.Done(): wg.Done() return default: p.poll(ctx, i+1) } } }(i) } wg.Wait() } ։啟ଟݸ1PMMFSฏߦ႔ཧ ։࢝ࣥߦ೚຿

Slide 13

Slide 13 text

'FUDIUIF/FX5BTL

Slide 14

Slide 14 text

ॳ࢝Խ࿈ઢ။۰౸ॄኄ໰୊

Slide 15

Slide 15 text

೗Ռ༗ݸ1PMMFS"HFOUॳ࢝Խ ݸ$MJFOU %%04

Slide 16

Slide 16 text

func (p *Poller) Poll(ctx context.Context, n int) { var wg sync.WaitGroup for i := 0; i < n; i++ { wg.Add(1) go func(i int) { for { select { case <-ctx.Done(): wg.Done() return default: p.poll(ctx, i+1) } } }(i) } wg.Wait() } ։啟ଟݸ1PMMFSฏߦ႔ཧ ։࢝ࣥߦ೚຿ ໰୊

Slide 17

Slide 17 text

೗Ռ༗ݸ1PMMFS"HFOUॳ࢝Խ ݸ$MJFOU %%04 㑌ݸॳ࢝Խ1PMMFSޙᬋػఀ಴YYYNT ՄҎݮ᫊%SPOF4FSWFSෛ፦

Slide 18

Slide 18 text

func (p *Poller) Poll(ctx context.Context, n int) { var wg sync.WaitGroup for i := 0; i < n; i++ { wg.Add(1) go func(i int) { for { select { case <-ctx.Done(): wg.Done() return default: p.poll(ctx, i+1) } } }(i) } wg.Wait() } ։啟ଟݸ1PMMFSฏߦ႔ཧ ։࢝ࣥߦ೚຿ ṜᬑՄҎᬋػఀ಴YYYNT

Slide 19

Slide 19 text

၊ઃਅ༗ݸ1PMMFSಉ࣌᪑ 4FSWFSᩇऔधཁࣥߦత೚຿ ᙛલᔒ༗೚Կ೚຿

Slide 20

Slide 20 text

func (p *Poller) Poll(ctx context.Context, n int) { var wg sync.WaitGroup for i := 0; i < n; i++ { wg.Add(1) go func(i int) { for { select { case <-ctx.Done(): wg.Done() return default: p.poll(ctx, i+1) } } }(i) } wg.Wait() } ։啟ଟݸ1PMMFSฏߦ႔ཧ ։࢝ࣥߦ೚຿ ໰୊

Slide 21

Slide 21 text

၊ઃਅ༗ݸ1PMMFSಉ࣌᪑ %SPOF4FSWFSᩇऔधཁࣥߦత೚຿ ᪑೭લղ๏Ұᒬɼࡏ1PMMFS׬੒ޙҰᒬ ᬋػఀ಴YYඵݮগ4FSWFSෛ፦

Slide 22

Slide 22 text

func (p *Poller) Poll(ctx context.Context, n int) { var wg sync.WaitGroup for i := 0; i < n; i++ { wg.Add(1) go func(i int) { for { select { case <-ctx.Done(): wg.Done() return default: p.poll(ctx, i+1) } } }(i) } wg.Wait() } ։啟ଟݸ1PMMFSฏߦ႔ཧ ։࢝ࣥߦ೚຿ ṜᬑՄҎᬋػఀ಴YYඵ

Slide 23

Slide 23 text

func (p *Poller) Poll(ctx context.Context, n int) { var wg sync.WaitGroup for i := 0; i < n; i++ { wg.Add(1) go func(i int) { for { select { case <-ctx.Done(): wg.Done() return default: p.poll(ctx, i+1) } } }(i) } wg.Wait() } ։啟ଟݸ1PMMFSฏߦ႔ཧ ։࢝ࣥߦ೚຿ ṜᬑՄҎᬋػఀ಴YYඵ ࠑղ๏။Ҿᚙॄኄ໰୊ʁ

Slide 24

Slide 24 text

4MFFQT 4MFFQT 4MFFQT େྔ5BTL༙ೖ

Slide 25

Slide 25 text

Drone Server 如何解決此問題

Slide 26

Slide 26 text

೗Կղܾେྔత"HFOU查ᨑ໨લ֘ࣥߦత೚຿

Slide 27

Slide 27 text

How to handle Agent request? • 建立 Timeout 為 30 秒的長連線 (Agent) • 使⽤ Go 語⾔ Map 建立所有 Agent 表列 • 使⽤ Channel 將項⽬分配到 Agent

Slide 28

Slide 28 text

func (s *Server) handleRequest(w http.ResponseWriter, r *http.Request) { ctx := r.Context() ctx, cancel := context.WithTimeout(ctx, defaultTimeout) defer cancel() in := &requestRequest{} err := json.NewDecoder(r.Body).Decode(in) if err != nil { writeBadRequest(w, err) return } stage, err := s.manager.Request(ctx, in.Request) if err != nil { writeError(w, err) return } json.NewEncoder(w).Encode(stage) } UJNF4FDPOE ݐཱඵత௕࿈ઢ

Slide 29

Slide 29 text

q.Lock() q.workers[w] = struct{}{} q.Unlock() select { case q.ready <- struct{}{}: default: } select { case <-ctx.Done(): q.Lock() delete(q.workers, w) q.Unlock() return nil, ctx.Err() case b := <-w.channel: return b, nil } ࣥߦҰ࣍查ᨑ%# "HFOUࢿ㘤์౸XPSLFS㚎 ᩇऔ4UBHFࢿ㘤 ௒աඵ௚઀ҠআXPSLFS

Slide 30

Slide 30 text

q.Lock() count := len(q.workers) pause := q.paused q.Unlock() if pause { return nil } if count == 0 { return nil } items, err := q.store.ListIncomplete(ctx) if err != nil { return err } ᩇऔॴ༗ঘະࣥߦత೚຿ Ꭴग़ॴ༗ະࣥߦత೚຿ฒ෼഑౸ಛఆత"HFOU

Slide 31

Slide 31 text

Drone Server 更好的解法?

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

㑌ݸ"HFOU୞။༗Ұᑍ࿈ઢ ՄҎಈଶௐ੔8PSLFSᏐྔ

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

֬อ୞༗ᄸҰ̣PEFՄҎ፤౸

Slide 37

Slide 37 text

ᩇऔ1JQFMJOF㑊ᱜ

Slide 38

Slide 38 text

؂ᡒ$BODFMࣄ݅

Slide 39

Slide 39 text

ctxdone, cancel := context.WithCancel(ctx) defer cancel() timeout := time.Duration(data.Repo.Timeout) * time.Minute ctxtimeout, cancel := context.WithTimeout(ctxdone, timeout) defer cancel() ctxcancel, cancel := context.WithCancel(ctxtimeout) defer cancel() go func() { done, _ := s.Client.Watch(ctxdone, data.Build.ID) if done { cancel() log.Debugln("received cancellation") } else { log.Debugln("done listening for cancellations") } }()

Slide 40

Slide 40 text

ctxdone, cancel := context.WithCancel(ctx) defer cancel() timeout := time.Duration(data.Repo.Timeout) * time.Minute ctxtimeout, cancel := context.WithTimeout(ctxdone, timeout) defer cancel() ctxcancel, cancel := context.WithCancel(ctxtimeout) defer cancel() go func() { done, _ := s.Client.Watch(ctxdone, data.Build.ID) if done { cancel() log.Debugln("received cancellation") } else { log.Debugln("done listening for cancellations") } }()

Slide 41

Slide 41 text

ctxdone, cancel := context.WithCancel(ctx) defer cancel() timeout := time.Duration(data.Repo.Timeout) * time.Minute ctxtimeout, cancel := context.WithTimeout(ctxdone, timeout) defer cancel() ctxcancel, cancel := context.WithCancel(ctxtimeout) defer cancel() go func() { done, _ := s.Client.Watch(ctxdone, data.Build.ID) if done { cancel() log.Debugln("received cancellation") } else { log.Debugln("done listening for cancellations") } }()

Slide 42

Slide 42 text

ctxdone, cancel := context.WithCancel(ctx) defer cancel() timeout := time.Duration(data.Repo.Timeout) * time.Minute ctxtimeout, cancel := context.WithTimeout(ctxdone, timeout) defer cancel() ctxcancel, cancel := context.WithCancel(ctxtimeout) defer cancel() go func() { done, _ := s.Client.Watch(ctxdone, data.Build.ID) if done { cancel() log.Debugln("received cancellation") } else { log.Debugln("done listening for cancellations") } }()

Slide 43

Slide 43 text

ctxdone, cancel := context.WithCancel(ctx) defer cancel() timeout := time.Duration(data.Repo.Timeout) * time.Minute ctxtimeout, cancel := context.WithTimeout(ctxdone, timeout) defer cancel() ctxcancel, cancel := context.WithCancel(ctxtimeout) defer cancel() go func() { done, _ := s.Client.Watch(ctxdone, data.Build.ID) if done { cancel() log.Debugln("received cancellation") } else { log.Debugln("done listening for cancellations") } }()

Slide 44

Slide 44 text

ctxdone, cancel := context.WithCancel(ctx) defer cancel() timeout := time.Duration(data.Repo.Timeout) * time.Minute ctxtimeout, cancel := context.WithTimeout(ctxdone, timeout) defer cancel() ctxcancel, cancel := context.WithCancel(ctxtimeout) defer cancel() go func() { done, _ := s.Client.Watch(ctxdone, data.Build.ID) if done { cancel() log.Debugln("received cancellation") } else { log.Debugln("done listening for cancellations") } }() 3FTQPOTF4UBUVT$PEF 3FTQPOTF4UBUVT$PEF

Slide 45

Slide 45 text

%JSFDUFE"DZDMJD(SBQI

Slide 46

Slide 46 text

%"(&YBNQMF

Slide 47

Slide 47 text

kind: pipeline type: docker name: default steps: - name: test image: golang:1.22 commands: - go build - go test -v - name: notify image: plugins/slack settings: channel: dev webhook: from_secret: endpoint ٬੡ԽჩᏐ */165@

Slide 48

Slide 48 text

END