Scale like a pro - Speaker Deck

Slide 1

Slide 1 text

Scale like a pro Distributed computing with message queues and Python Roman Imankulov | Python Porto | December 2017

Slide 2

Slide 2 text

Source: https://blog.kissmetrics.com/wp-content/uploads/2011/04/loading-time.pdf

Slide 3

Slide 3 text

browser ./webserver.py request POST /obj/ obj.update()

Slide 4

Slide 4 text

browser ./webserver.py request POST /obj/ obj.update() update search indexes

Slide 5

Slide 5 text

browser ./webserver.py request POST /obj/ obj.update() update search indexes send email “object updated” …

Slide 6

Slide 6 text

browser ./webserver.py request POST /obj/ obj.update() update search indexes update business analytics send email “object updated” …

Slide 7

Slide 7 text

browser ./webserver.py request POST /obj/ obj.update() update search indexes update business analytics send email “object updated” … response

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

browser ./webserver.py request POST /obj/ response obj.update() ./worker.py ./worker.py ./worker.py update search indexes send email “object updated” update business analytics

Slide 10

Slide 10 text

Message queues

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

queue

Slide 14

Slide 14 text

frontends queue

Slide 15

Slide 15 text

frontends queue workers

Slide 16

Slide 16 text

frontends queue job 1 workers

Slide 17

Slide 17 text

frontends queue job 1 workers job 2

Slide 18

Slide 18 text

frontends queue job 1 workers job 2 job 3

Slide 19

Slide 19 text

frontends queue put() job 1 workers job 2 job 3

Slide 20

Slide 20 text

frontends queue put() job 1 workers job 2 job 3

Slide 21

Slide 21 text

frontends queue job 1 workers job 2 job 3

Slide 22

Slide 22 text

frontends queue job 1 workers put() job 2 job 3

Slide 23

Slide 23 text

frontends queue job 1 workers put() job 2 job 3

Slide 24

Slide 24 text

frontends queue job 1 workers job 2 job 3

Slide 25

Slide 25 text

frontends queue job 1 workers put() job 2 job 3

Slide 26

Slide 26 text

frontends queue job 1 workers put() job 2 job 3

Slide 27

Slide 27 text

frontends queue job 1 workers job 2 job 3

Slide 28

Slide 28 text

frontends queue job 1 get() workers job 2 job 3

Slide 29

Slide 29 text

frontends queue get() workers job 2 job 3 job 1

Slide 30

Slide 30 text

frontends queue workers job 2 job 3 job 1

Slide 31

Slide 31 text

frontends queue workers get() job 2 job 3 job 1

Slide 32

Slide 32 text

frontends queue workers get() job 3 job 1 job 2

Slide 33

Slide 33 text

frontends queue workers job 3 job 1 job 2

Slide 34

Slide 34 text

frontends queue workers job 3 job 2

Slide 35

Slide 35 text

frontends queue workers job 3 job 2 get()

Slide 36

Slide 36 text

frontends queue workers job 2 job 3 get()

Slide 37

Slide 37 text

frontends queue workers job 2 job 3

Slide 38

Slide 38 text

frontends queue workers

Slide 39

Slide 39 text

Queue in python

Slide 40

Slide 40 text

Multiprocessing

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

Celery The queue out of the box

Slide 49

Slide 49 text

No content

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

Celery Workﬂows chains, groups and chords

Slide 55

Slide 55 text

Task signatures

Slide 56

Slide 56 text

Task signatures

Slide 57

Slide 57 text

Task signatures

Slide 58

Slide 58 text

Chains a(…) b(…) c(…)

Slide 59

Slide 59 text

Chains a(…) b(…) c(…)

Slide 60

Slide 60 text

Chains a(…) b(…) c(…)

Slide 61

Slide 61 text

a(…) b(…) c(…) Groups

Slide 62

Slide 62 text

a(…) b(…) c(…) Groups

Slide 63

Slide 63 text

a(…) b(…) c(…) Groups

Slide 64

Slide 64 text

Chords a(…) b(…) c(…) d(…)

Slide 65

Slide 65 text

Chords a(…) b(…) c(…) d(…)

Slide 66

Slide 66 text

Chords a(…) b(…) c(…) d(…)

Slide 67

Slide 67 text

Celery Extras out of the box

Slide 68

Slide 68 text

No content

Slide 69

Slide 69 text

• Diﬀerent backends

Slide 70

Slide 70 text

• Diﬀerent backends • Diﬀerent serializers

Slide 71

Slide 71 text

• Diﬀerent backends • Diﬀerent serializers • Callbacks / errbacks

Slide 72

Slide 72 text

• Diﬀerent backends • Diﬀerent serializers • Callbacks / errbacks • Progress reports from tasks

Slide 73

Slide 73 text

• Diﬀerent backends • Diﬀerent serializers • Callbacks / errbacks • Progress reports from tasks • Delayed tasks

Slide 74

Slide 74 text

• Diﬀerent backends • Diﬀerent serializers • Callbacks / errbacks • Progress reports from tasks • Delayed tasks • Ignored results

Slide 75

Slide 75 text

• Diﬀerent backends • Diﬀerent serializers • Callbacks / errbacks • Progress reports from tasks • Delayed tasks • Ignored results • Expiring results

Slide 76

Slide 76 text

• Diﬀerent backends • Diﬀerent serializers • Callbacks / errbacks • Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies

Slide 77

Slide 77 text

Slide 78

Slide 78 text

• Diﬀerent backends • Diﬀerent serializers • Callbacks / errbacks • Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute)