Tweet
Tweet

Slide 1

Slide 1 text

Job Queue in Golang Bo-Yi Wu 2019.10.19

Slide 2

Slide 2 text

About me • Software Engineer in Mediatek • Member of Drone CI/CD Platform • Member of Gitea Platform • Member of Gin Golang Framework • Maintain Some GitHub Actions Plugins. • Teacher of Udemy Platform: Golang + Drone

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Queue (Open Source) 3BCCJU.2 /42

Slide 5

Slide 5 text

Why not use Open Source?

Slide 6

Slide 6 text

Single Service • Build in SQLite (LDAP, Job List) • Build in Cache (LRU) • Build for single binary • No third-party dependency

Slide 7

Slide 7 text

Before talking about Job Queue

Slide 8

Slide 8 text

Buffered vs Unbuffered http://bit.ly/understand-channel

Slide 9

Slide 9 text

Goroutine func main() { go func() { fmt.Println("GO GO GO") }() time.Sleep(1 * time.Second) }

Slide 10

Slide 10 text

Unbuffered make(chan bool)

Slide 11

Slide 11 text

Unbuffered Channel func main() { c := make(chan bool) go func() { fmt.Println("GO GO GO") c <- true }() <-c }

Slide 12

Slide 12 text

Unbuffered Channel func main() { c := make(chan bool) go func() { fmt.Println("GO GO GO") c <- true }() <-c }

Slide 13

Slide 13 text

Unbuffered Channel func main() { c := make(chan bool) go func() { fmt.Println("GO GO GO") <-c }() c <- true }

Slide 14

Slide 14 text

func main() { c := make(chan bool) go func() { fmt.Println("GO GO GO") c <- true c <- true }() <-c time.Sleep(1 * time.Second) } Unbuffered Channel

Slide 15

Slide 15 text

buffered make(chan bool, 1)

Slide 16

Slide 16 text

Buffered channel func main() { c := make(chan bool, 1) go func() { fmt.Println("GO GO GO") <-c }() c <- true }

Slide 17

Slide 17 text

Buffered channel func main() { c := make(chan bool, 1) go func() { fmt.Println("GO GO GO") <-c }() c <- true }

Slide 18

Slide 18 text

Buffered channel func main() { c := make(chan bool, 1) go func() { fmt.Println("GO GO GO") c <- true }() <-c }

Slide 19

Slide 19 text

How to implement Job Queue in Go

Slide 20

Slide 20 text

Sometimes you don’t need A job queue go process("job01")

Slide 21

Slide 21 text

func worker(jobChan <-chan Job) { for job := range jobChan { process(job) } } // make a channel with a capacity of 1024. jobChan := make(chan Job, 1024) // start the worker go worker(jobChan) // enqueue a job jobChan <- job

Slide 22

Slide 22 text

func worker(jobChan <-chan Job) { for job := range jobChan { process(job) } } // make a channel with a capacity of 1024. jobChan := make(chan Job, 1024) // start the worker go worker(jobChan) // enqueue a job jobChan <- job

Slide 23

Slide 23 text

Block if there already are 1024 jobs jobChan := make(chan Job, 1024)

Slide 24

Slide 24 text

Enqueue without blocking

Slide 25

Slide 25 text

func Enqueue(job Job, jobChan chan<- Job) bool { select { case jobChan <- job: return true default: return false } } if !Enqueue(job, job100) { Error( http.StatusServiceUnavailable, "max capacity reached", ) return }

Slide 26

Slide 26 text

Stopping the worker?

Slide 27

Slide 27 text

func main() { ch := make(chan int, 2) go func() { ch <- 1 ch <- 2 }() for n := range ch { fmt.Println(n) } }

Slide 28

Slide 28 text

func main() { ch := make(chan int, 2) go func() { ch <- 1 ch <- 2 close(ch) }() for n := range ch { fmt.Println(n) } }

Slide 29

Slide 29 text

func main() { ch := make(chan int, 2) go func() { ch <- 1 ch <- 2 }() go func() { for n := range ch { fmt.Println(n) } }() time.Sleep(1 * time.Second) }

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

Setup Consumer

Slide 32

Slide 32 text

type Consumer struct { inputChan chan int jobsChan chan int } const PoolSize = 2 func main() { // create the consumer consumer := Consumer{ inputChan: make(chan int, 1), jobsChan: make(chan int, PoolSize), } }

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

func (c *Consumer) queue(input int) { fmt.Println("send input value:", input) c.jobsChan <- input } func (c *Consumer) worker(num int) { for job := range c.jobsChan { fmt.Println("worker:", num, " job value:", job) } } for i := 0; i < PoolSize; i++ { go consumer.worker(i) }

Slide 35

Slide 35 text

consumer.queue(1) consumer.queue(2) consumer.queue(3) consumer.queue(4) blocking by poll size = 2

Slide 36

Slide 36 text

Output (Poll Size = 2) send input value: 1 send input value: 2 send input value: 3 worker: 0 job value: 1 send input value: 4 worker: 0 job value: 2 worker: 0 job value: 3 worker: 0 job value: 4

Slide 37

Slide 37 text

rewrite queue func func (c *Consumer) queue(input int) bool { fmt.Println("send input value:", input) select { case c.jobsChan <- input: return true default: return false } }

Slide 38

Slide 38 text

Output (Poll Size = 2) send input value: 1 send input value: 2 send input value: 3 send input value: 4 worker: 0 job value: 1 worker: 0 job value: 2

Slide 39

Slide 39 text

Shutdown with Sigterm Handling

Slide 40

Slide 40 text

func WithContextFunc(ctx context.Context, f func()) context.Context { ctx, cancel := context.WithCancel(ctx) go func() { c := make(chan os.Signal) signal.Notify(c, syscall.SIGINT, syscall.SIGTERM) defer signal.Stop(c) select { case <-ctx.Done(): case <-c: f() cancel() } }() return ctx }

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

func WithContextFunc(ctx context.Context, f func()) context.Context { ctx, cancel := context.WithCancel(ctx) go func() { c := make(chan os.Signal) signal.Notify(c, syscall.SIGINT, syscall.SIGTERM) defer signal.Stop(c) select { case <-ctx.Done(): case <-c: f() cancel() } }() return ctx }

Slide 43

Slide 43 text

func (c Consumer) startConsumer(ctx context.Context) { for { select { case job := <-c.inputChan: if ctx.Err() != nil { close(c.jobsChan) return } c.jobsChan <- job case <-ctx.Done(): close(c.jobsChan) return } } }

Slide 44

Slide 44 text

Cancel by ctx.Done() event func (c *Consumer) worker(num int) { for job := range c.jobsChan { fmt.Println("worker:", num, " job value:", job) } }

Slide 45

Slide 45 text

Canceling Workers without Context

Slide 46

Slide 46 text

cancelChan := make(chan struct{}) go worker(jobChan, cancelChan) func worker(jobChan <-chan Job, cancelChan <-chan struct{}) { for { select { case <-cancelChan: return case job := <-jobChan: process(job) } } } // to cancel the worker, close the cancel channel close(cancelChan) Create a cancel channel

Slide 47

Slide 47 text

cancelChan := make(chan struct{}) go worker(jobChan, cancelChan) func worker(jobChan <-chan Job, cancelChan <-chan struct{}) { for { select { case <-cancelChan: return case job := <-jobChan: process(job) } } } // to cancel the worker, close the cancel channel close(cancelChan) Create a cancel channel close(cancelChan)

Slide 48

Slide 48 text

Graceful shutdown with worker sync.WaitGroup

Slide 49

Slide 49 text

No content

Slide 50

Slide 50 text

wg := &sync.WaitGroup{} wg.Add(numberOfWorkers) // Start [PoolSize] workers for i := 0; i < PoolSize; i++ { go consumer.worker(i) }

Slide 51

Slide 51 text

WaitGroup WaitGroup WaitGroup WaitGroup

Slide 52

Slide 52 text

func (c Consumer) worker(wg *sync.WaitGroup) { defer wg.Done() for job := range c.jobsChan { // handle the job event } }

Slide 53

Slide 53 text

Add WaitGroup after Cancel Function

Slide 54

Slide 54 text

func WithContextFunc(ctx context.Context, f func()) context.Context { ctx, cancel := context.WithCancel(ctx) go func() { c := make(chan os.Signal) signal.Notify(c, syscall.SIGINT, syscall.SIGTERM) defer signal.Stop(c) select { case <-ctx.Done(): case <-c: cancel() f() } }() return ctx } Add WaitGroup after Cancel Function

Slide 55

Slide 55 text

wg := &sync.WaitGroup{} wg.Add(numberOfWorkers) ctx := signal.WithContextFunc( context.Background(), func() { wg.Wait() close(finishChan) }, ) go consumer.startConsumer(ctx)

Slide 56

Slide 56 text

End of Program select { case <-finished: case err := <-errChannel: if err != nil { return err } }

Slide 57

Slide 57 text

How to auto-scaling build agent?

Slide 58

Slide 58 text

No content

Slide 59

Slide 59 text

Communicate between server and agent

Slide 60

Slide 60 text

No content

Slide 61

Slide 61 text

Jobs Schema

Slide 62

Slide 62 text

r := e.Group("/rpc") r.Use(rpc.Check()) { r.POST("/v1/healthz", web.RPCHeartbeat) r.POST("/v1/request", web.RPCRquest) r.POST("/v1/accept", web.RPCAccept) r.POST("/v1/details", web.RPCDetails) r.POST("/v1/updateStatus", web.RPCUpdateStatus) r.POST("/v1/upload", web.RPCUploadBytes) r.POST("/v1/reset", web.RPCResetStatus) } Check RPC Secret

Slide 63

Slide 63 text

/rpc/v1/accept Update jobs set version = (oldVersion + 1) where machine = "fooBar" and version = oldVersion

Slide 64

Slide 64 text

Create multiple worker

Slide 65

Slide 65 text

if r.Capacity != 0 { var g errgroup.Group for i := 0; i < r.Capacity; i++ { g.Go(func() error { return r.start(ctx, 0) }) time.Sleep(1 * time.Second) } return g.Wait() }

Slide 66

Slide 66 text

Break for and select loop func (r *Runner) start(ctx context.Context, id int64) error { LOOP: for { select { case <-ctx.Done(): return ctx.Err() default: r.poll(ctx, id) if r.Capacity == 0 { break LOOP } } time.Sleep(1 * time.Second) } return nil }

Slide 67

Slide 67 text

How to cancel the current Job?

Slide 68

Slide 68 text

No content

Slide 69

Slide 69 text

No content

Slide 70

Slide 70 text

No content

Slide 71

Slide 71 text

Context with Cancel or Timeout ctx, cancel := context.WithCancel(context.Background()) defer cancel() timeout, cancel := context.WithTimeout(ctx, 60*time.Minute) defer cancel() Job03 context

Slide 72

Slide 72 text

Context with Cancel or Timeout ctx, cancel := context.WithCancel(context.Background()) defer cancel() timeout, cancel := context.WithTimeout(ctx, 60*time.Minute) defer cancel() Job03 context Job05 context

Slide 73

Slide 73 text

Watch the Cancel event go func() { done, _ := r.Manager.Watch(ctx, id) if done { cancel() } }()

Slide 74

Slide 74 text

Handle cancel event on Server subscribers: make(map[chan struct{}]int64), cancelled: make(map[int64]time.Time),

Slide 75

Slide 75 text

User cancel running job c.Lock() c.cancelled[id] = time.Now().Add(time.Minute * 5) for subscriber, build := range c.subscribers { if id == build { close(subscriber) } } c.Unlock()

Slide 76

Slide 76 text

Agent subscribe the cancel event for { select { case <-ctx.Done(): return false, ctx.Err() case <-time.After(time.Minute): c.Lock() _, ok := c.cancelled[id] c.Unlock() if ok { return true, nil } case <-subscriber: return true, nil } }

Slide 77

Slide 77 text

case <-time.After(time.Minute): c.Lock() _, ok := c.cancelled[id] c.Unlock() if ok { return true, nil }

Slide 78

Slide 78 text

case <-time.After(time.Minute): c.Lock() _, ok := c.cancelled[id] c.Unlock() if ok { return true, nil } 1 Cancel

Slide 79

Slide 79 text

case <-time.After(time.Minute): c.Lock() _, ok := c.cancelled[id] c.Unlock() if ok { return true, nil } 1 2 Reconnect Server Cancel

Slide 80

Slide 80 text

https://www.udemy.com/course/golang-fight/?couponCode=GOLANG201911

Slide 81

Slide 81 text

https://www.udemy.com/course/devops-oneday/?couponCode=DEVOPS201911

Slide 82

Slide 82 text

Any Question?