Practical Celery

Lessons from using Celery tasks at scale in a Django application

Transcript

1. PRACTICAL CELERY
2. CAMERON MASKE twitter: @cameronmaske email: cam@trackmaven.com web: http://cameronmaske.com
3. WHAT WE'LL COVER...
4. WHAT IS CELERY? HOW DOES IT WORK?
5. USING CELERY, BEST PRACTICES AND SCALING.
6. SURVEY
7. CELERY ASYNCHRONOUS DISTRIBUTED TASK QUEUE
8. OUT OF THE REQUEST/RESPONSE CYCLE. Example: Sending emails asynchronously.
9. TASKS IN THE BACKGROUND. Example: Computational heavy jobs. Example: Interacting with external APIs.
10. PERIODIC JOBS.
11. HISTORY Python. Released (0.1) in 2009. Currently on 3.1, with 3.2 in alpha. Developed by Ask Solem (@asksol)
12. ARCHITECTURE
13. PRODUCER Produces a task for the queue.
14. BROKER Stores the task backlog Answers, what work remains to be done? RabbitMQ, Redis, SQLAlchemy, Django's ORM, MongoDB...
15. WORKER Execute and consumes tasks. Distributed.
16. RESULTS BACKEND. Stores the results from our tasks. Redis, Redis, SQLAlchemy, Django's ORM, MongoDB... Optional!
17. EXAMPLE
18. f r o m c e l e r y i m p o r t C e l e r y a p p = C e l e r y ( ' t a s k s ' , b a c k e n d = ' a m q p ' , b r o k e r = ' a m q p : / / g u e s t @ l o c a l h o s t / / ' ) @ a p p . t a s k d e f a d d ( x , y ) : r e t u r n x + y
19. > > > r e s u l t = a d d . d e l a y ( 4 , 4 ) > > > r e s u l t . s t a t e ' S U C C E S S ' > > > r e s u l t . i d ' 4 c c 7 4 3 8 e - a f d 4 - 4 f 8 f - a 2 f 3 - f 4 6 5 6 7 e 7 c a 7 7 ' > > > r e s u l t . g e t ( ) 8 http://celery.readthedocs.org/en/latest/reference/celery.result.html
20. PICK YOUR FLAVOR. @ a p p . t a s k d e f a d d ( x , y ) : r e t u r n x + y a d d ( 2 , 4 ) c l a s s A d d T a s k ( a p p . T a s k ) : d e f r u n ( s e l f , x , y ) : r e t u r n x + y A d d T a s k ( ) . r u n ( 2 , 4 )
21. # A s y n c a d d . d e l a y ( 2 , 4 ) a d d . a p p l y _ a y n c ( a r g s = ( 2 , 4 ) , e x p i r e s = 3 0 ) # E a g e r ! r e s u l t = a d d . a p p l y ( a r g s = ( 2 , 4 ) ) # E x e c u t e s l o c a l l y . # O r . . . a d d ( 2 , 4 ) # D o e s n o t r e t u r n a c e l e r y r e s u l t !
22. INTERGRATING WITH DJANGO.
23. BEWARE OF D J A N G O - C E L E R Y .
24. http://docs.celeryproject.org/en/master/django/first-steps-with- django.html - p r o j e c t / - c o n f i g / _ _ i n i t _ _ . p y - c o n f i g / s e t t i n g s . p y - c o n f i g / u r l s . p y - m a n a g e . p y
25. # p r o j e c t / c o n f i g / c e l e r y . p y f r o m _ _ f u t u r e _ _ i m p o r t a b s o l u t e _ i m p o r t i m p o r t o s f r o m c e l e r y i m p o r t C e l e r y f r o m d j a n g o . c o n f i m p o r t s e t t i n g s # S e t t h e d e f a u l t D j a n g o s e t t i n g s m o d u l e f o r t h e ' c e l e r y ' p r o g r a m . o s . e n v i r o n . s e t d e f a u l t ( ' D J A N G O _ S E T T I N G S _ M O D U L E ' , ' c o n f i g . s e t t i n g s ' ) a p p = C e l e r y ( ' a p p ' ) # U s i n g a s t r i n g h e r e m e a n s t h e w o r k e r w i l l n o t h a v e t o # p i c k l e t h e o b j e c t w h e n u s i n g W i n d o w s . a p p . c o n f i g _ f r o m _ o b j e c t ( ' d j a n g o . c o n f : s e t t i n g s ' ) a p p . a u t o d i s c o v e r _ t a s k s ( l a m b d a : s e t t i n g s . I N S T A L L E D _ A P P S ) @ a p p . t a s k ( b i n d = T r u e ) d e f d e b u g _ t a s k ( s e l f ) : p r i n t ( ' R e q u e s t : { 0 ! r } ' . f o r m a t ( s e l f . r e q u e s t ) )
26. # p r o j e c t / c o n f i g / _ _ i n i t _ _ . p y f r o m _ _ f u t u r e _ _ i m p o r t a b s o l u t e _ i m p o r t # T h i s w i l l m a k e s u r e t h e a p p i s a l w a y s i m p o r t e d w h e n # D j a n g o s t a r t s s o t h a t s h a r e d _ t a s k w i l l u s e t h i s a p p . f r o m . c e l e r y i m p o r t a p p a s c e l e r y _ a p p _ _ a l l _ _ = [ ' c e l e r y _ a p p ' ]
27. c e l e r y - A p r o j e c t w o r k e r - l i n f o
28. TESTING # s e t t i n g s . p y i m p o r t s y s i f ' t e s t ' i n s y s . a r g v : C E L E R Y _ E A G E R _ P R O P A G A T E S _ E X C E P T I O N S = T r u e , C E L E R Y _ A L W A Y S _ E A G E R = T r u e , B R O K E R _ B A C K E N D = ' m e m o r y '
29. PATTERNS AND BEST PRACTICES.
30. NEVER PASS OBJECTS AS ARGUMENTS.
31. # B a d @ a p p . t a s k ( ) d e f s e n d _ r e m i n d e r ( r e m i n d e r ) : r e m i n d e r . s e n d _ e m a i l ( ) # G o o d @ a p p . t a s k ( ) d e f s e n d _ r e m i n d e r ( p k ) : t r y : r e m i n d e r = R e m i n d e r . o b j e c t s . g e t ( p k = p k ) e x c e p t R e m i n d e r . D o e s N o t E x i s t : r e t u r n r e m i n d e r . s e n d _ e m a i l ( )
32. KEEP TASKS GRANUAL. CAN PROCESS MORE IN PARALLEL.
33. AVOID LAUNCHING SYNCHRONOUS SUBTASKS
34. # B a d @ a p p . t a s k d e f u p d a t e _ p a g e _ i n f o ( u r l ) : p a g e = f e t c h _ p a g e . d e l a y ( u r l ) . g e t ( ) i n f o = p a r s e _ p a g e . d e l a y ( u r l , p a g e ) . g e t ( ) s t o r e _ p a g e _ i n f o . d e l a y ( u r l , i n f o ) @ a p p . t a s k d e f f e t c h _ p a g e ( u r l ) : r e t u r n m y h t t p l i b . g e t ( u r l ) @ a p p . t a s k d e f p a r s e _ p a g e ( u r l , p a g e ) : r e t u r n m y p a r s e r . p a r s e _ d o c u m e n t ( p a g e ) @ a p p . t a s k d e f s t o r e _ p a g e _ i n f o ( u r l , i n f o ) : r e t u r n P a g e I n f o . o b j e c t s . c r e a t e ( u r l , i n f o )
35. # G o o d d e f u p d a t e _ p a g e _ i n f o ( u r l ) : c h a i n = f e t c h _ p a g e . s ( ) | p a r s e _ p a g e . s ( ) | s t o r e _ p a g e _ i n f o . s ( u r l ) c h a i n ( ) @ a p p . t a s k ( ) d e f f e t c h _ p a g e ( u r l ) : r e t u r n m y h t t p l i b . g e t ( u r l ) @ a p p . t a s k ( ) d e f p a r s e _ p a g e ( p a g e ) : r e t u r n m y p a r s e r . p a r s e _ d o c u m e n t ( p a g e ) @ a p p . t a s k ( i g n o r e _ r e s u l t = T r u e ) d e f s t o r e _ p a g e _ i n f o ( i n f o , u r l ) : P a g e I n f o . o b j e c t s . c r e a t e ( u r l = u r l , i n f o = i n f o ) http://celery.readthedocs.org/en/latest/userguide/canvas.html
36. PERIODIC TASKS. http://celery.readthedocs.org/en/latest/userguide/periodic- tasks.html
37. f r o m d a t e t i m e i m p o r t t i m e d e l t a @ a p p . p e r i o d i c _ t a s k ( r u n _ e v e r y = t i m e d e l t a ( m i n u t e s = 5 ) ) : d e f r u n _ e v e r y _ f i v e ( ) : p a s s
38. f r o m d a t e t i m e i m p o r t t i m e d e l t a c l a s s R u n E v e r y F i v e ( a p p . P e r i o d i c T a s k ) : r u n _ e v e r y = t i m e d e l t a ( m i n u t e s = 5 ) d e f r u n ( s e l f ) : p a s s
39. f r o m d a t e t i m e i m p o r t t i m e d e l t a @ a p p . t a s k ( ) : d e f r u n _ e v e r y _ f i v e ( ) : p a s s C E L E R Y B E A T _ S C H E D U L E = { ' r u n - e v e r y - f i v e ' : { ' t a s k ' : ' t a s k s . r u n _ e v e r y _ f i v e ' , ' s c h e d u l e ' : t i m e d e l t a ( s e c o n d s = 3 0 ) } , }
40. CRON STYLE. f r o m c e l e r y . s c h e d u l e s i m p o r t c r o n t a b c r o n t a b ( m i n u t e = 0 , h o u r = ' * / 3 ' ) # E v e r y 3 h o u r s . c r o n t a b ( d a y _ o f _ w e e k = ' s u n d a y ' ) # E v e r y m i n u t e o n S u n d a y s . c r o n t a b ( 0 , 0 , 0 , m o n t h _ o f _ y e a r = ' * / 3 ' ) # F i r s t m o n t h o f e v e r y q u a r t e r .
41. @ a p p . p e r i o d i c _ t a s k ( r u n _ e v e r y = c r o n t a b ( m i n u t e = 0 , h o u r = 1 ) ) d e f s c h e d u l e _ e m a i l s ( ) : u s e r _ i d s = U s e r . o b j e c t s . v a l u e s _ l i s t ( ' i d ' , f l a t = T r u e ) f o r u s e r _ i d i n u s e r _ i d s : s e n d _ d a i l y _ e m a i l . d e l a y ( u s e r _ i d ) @ a p p . t a s k ( ) d e f s e n d _ d a i l y _ e m a i l ( u s e r _ i d ) : u s e r = U s e r . o b j e c t s . g e t ( i d = u s e r _ i d ) t r y : t o d a y = d a t e t i m e . n o w ( ) E m a i l . o b j e c t s . g e t ( u s e r = u s e r , d a t e _ _ y e a r = t o d a y . y e a r , d a t e _ _ m o n t h = t o d a y . m o n t h , d a t e _ _ d a y = t o d a y . d a y e x c e p t E m a i l . D o e s N o t E x i s t : e m a i l = E m a i l ( u s e r = u s e r , b o d y = " H e y , d o n ' t f o r g e t t o L O G I N P L E A S E ! " ) e m a i l . s e n d ( ) e m a i l . s a v e ( )
42. CELERY BEAT A.K.A THE SCHEDULER. c e l e r y - A p r o j e c t b e a t
43. NEVER RUN A BEAT + WORKER ON A SINGLE CELERY PROCESS. # R e a l l y b a d i d e a . . . . c e l e r y - A p r o j e c t w o r k e r - B
44. FREQUENTLY RUNNING PERIODIC TASKS. BEWARE OF "TASK STACKING"
45. Schedule task runs every 5 minutes. Tasks take 30 minutes. Schedule task stacks. Bad stuff.
46. EXPIRES! f r o m t i m e i m p o r t s l e e p @ a p p . p e r i o d i c _ t a s k ( e x p i r e s = 5 * 6 0 , r u n _ e v e r y = t i m e d e l t a ( m i n u t e s = 5 ) ) d e f s c h e d u l e _ t a s k ( ) : f o r _ i n r a n g e ( 3 0 ) : o n e _ m i n u t e _ t a s k . d e l a y ( ) @ a p p . t a s k ( e x p i r e s = 5 * 6 0 ) d e f o n e _ m i n u t e _ t a s k ( ) : s l e e p ( 6 0 )
47. THINGS GO WRONG IN TASKS!
48. RETRY
49. f r o m c e l e r y . e x c e p t i o n s i m p o r t R e t r y @ a p p . t a s k ( m a x _ r e t r i e s = 1 0 ) d e f g a t h e r _ d a t a ( ) : t r y : d a t a = a p i . g e t _ d a t a ( ) # e t c , e t c , . . . e x c e p t a p i . R a t e L i m i t e d a s e : r a i s e R e t r y ( e x c = e , w h e n = e . c o o l d o w n ) e x c e p t a p i . I s D o w n : r e t u r n
50. ERROR INSIGHT.
51. SENTRY.
52. STAGES
53. c l a s s D e b u g T a s k ( a p p . T a s k ) : d e f a f t e r _ r e t u r n ( s e l f , s t a t u s , r e t v a l , t a s k _ i d , a r g s , k w a r g s , e i n f o ) : p r i n t ( " I ' m d o n e ! " ) d e f o n _ f a i l u r e ( s e l f , e x c , t a s k _ i d , a r g s , k w a r g s , e i n f o ) : p r i n t ( " I f a i l e d : ( " ) d e f o n _ r e t r y ( s e l f , e x c , t a s k _ i d , a r g s , k w a r g s , e i n f o ) : p r i n t ( " I ' l l t r y a g a i n ! " ) d e f o n _ s u c c e s s ( s e l f , r e t v a l , t a s k _ i d , a r g s , k w a r g s ) : p r i n t ( " I d i d i t ! " )
54. ABSTRACT c l a s s A b s t r a c t T a s k ( a p p . T a s k ) : a b s t r a c t = T r u e d e f a f t e r _ r e t u r n ( s e l f , * a r g s , * * k w a r g s ) : p r i n t ( " A l l d o n e ! " ) @ a p p . t a s k ( b a s e = A b s t r a c t T a s k ) d e f a d d ( x , y ) : r e t u r n x + y
55. INSTANTIATION c l a s s D a t a b a s e T a s k ( a p p . T a s k ) : a b s t r a c t = T r u e _ d b = N o n e @ p r o p e r t y d e f d b ( s e l f ) : i f s e l f . _ d b i s N o n e : s e l f . _ d b = D a t a b a s e . c o n n e c t ( ) r e t u r n s e l f . _ d b
56. ENSURE A TASK IS EXECUTED ONE AT A TIME
57. f r o m c e l e r y i m p o r t t a s k f r o m c e l e r y . u t i l s . l o g i m p o r t g e t _ t a s k _ l o g g e r f r o m d j a n g o . c o r e . c a c h e i m p o r t c a c h e f r o m d j a n g o . u t i l s . h a s h c o m p a t i m p o r t m d 5 _ c o n s t r u c t o r a s m d 5 f r o m d j a n g o f e e d s . m o d e l s i m p o r t F e e d l o g g e r = g e t _ t a s k _ l o g g e r ( _ _ n a m e _ _ ) L O C K _ E X P I R E = 6 0 * 5 # L o c k e x p i r e s i n 5 m i n u t e s @ t a s k d e f i m p o r t _ f e e d ( f e e d _ u r l ) : # T h e c a c h e k e y c o n s i s t s o f t h e t a s k n a m e a n d t h e M D 5 d i g e s t # o f t h e f e e d U R L . f e e d _ u r l _ d i g e s t = m d 5 ( f e e d _ u r l ) . h e x d i g e s t ( ) l o c k _ i d = ' { 0 } - l o c k - { 1 } ' . f o r m a t ( s e l f . n a m e , f e e d _ u r l _ h e x d i g e s t ) # c a c h e . a d d f a i l s i f i f t h e k e y a l r e a d y e x i s t s a c q u i r e _ l o c k = l a m b d a : c a c h e . a d d ( l o c k _ i d , ' t r u e ' , L O C K _ E X P I R E ) # m e m c a c h e d e l e t e i s v e r y s l o w , b u t w e h a v e t o u s e i t t o t a k e # a d v a n t a g e o f u s i n g a d d ( ) f o r a t o m i c l o c k i n g r e l e a s e _ l o c k = l a m b d a : c a c h e . d e l e t e ( l o c k _ i d ) l o g g e r . d e b u g ( ' I m p o r t i n g f e e d : % s ' , f e e d _ u r l ) i f a c q u i r e _ l o c k ( ) : t r y : f e e d = F e e d . o b j e c t s . i m p o r t _ f e e d ( f e e d _ u r l ) f i n a l l y : r e l e a s e _ l o c k ( ) r e t u r n f e e d . u r l l o g g e r . d e b u g ( ' F e e d % s i s a l r e a d y b e i n g i m p o r t e d b y a n o t h e r w o r k e r ' , f e e d _ u r l )
58. IMPORTANT SETTINGS
59. # s e t t i n g s . p y C E L E R Y _ I G N O R E _ R E S U L T = T r u e C E L E R Y D _ T A S K _ S O F T _ T I M E _ L I M I T = 5 0 0 C E L E R Y D _ T A S K _ T I M E _ L I M I T = 1 0 0 0
60. # t a s k s . p y a p p . t a s k ( i g n o r e _ r e s u l t = T r u e , s o f t _ t i m e _ l i m i t = 6 0 , t i m e _ l i m i t = 1 2 0 ) d e f a d d ( x , y ) : p a s s
61. # s e t t i n g s . p y C E L E R Y D _ M A X _ T A S K S _ P E R _ C H I L D = 5 0 0 C E L E R Y D _ P R E F E T C H _ M U L T I P L I E R = 4
62. BROKER
63. SO MANY CHOICES! RabbitMQ Redis SQLAlchemy Django's ORM MongoDB Amazon SQS CouchDB Beanstalk IronMQ
64. DJANGO ORM. # s e t t i n g s . p y B R O K E R _ U R L = ' d j a n g o : / / ' I N S T A L L E D _ A P P S = ( ' k o m b u . t r a n s p o r t . d j a n g o ' , ) C E L E R Y _ R E S U L T _ B A C K E N D = ' d j c e l e r y . b a c k e n d s . d a t a b a s e : D a t a b a s e B a c k e n d ' p y t h o n m a n a g e . p y s y n c d b
65. DON'T DO THIS FOR ANYTHING SERIOUS.
66. USE RABBITMQ
67. C OPTIMIZED LIBRARY $ p i p i n s t a l l l i b r a b b i t m q
68. WORKERS
69. CONCURRENCY c e l e r y - A p r o j e c t w o r k e r - C 1 0 c e l e r y - A p r o j e c t w o r k e r - - a u t o s c a l e = 1 0 , 1
70. INCREASED CONCURRENCY CAN QUICKLY DRAIN CONNECTIONS ON YOUR DATABASE Use a connection pooler (pgbouncer).
71. ROUTING
72. C E L E R Y _ R O U T E S = { ' e m a i l . t a s k s . s e n d _ m a i l ' : { ' q u e u e ' : ' p r i o r i t y ' , } , } # o r s e n d _ m a i l . a p p l y _ a s y n c ( q u e u e = " p r i o r i t y " ) c e l e r y - A p r o j e c t w o r k e r - Q e m a i l
73. DEDICATED WORKERS.
74. BOTTLENECKS
75. Identify Fix Repeat
76. Make tasks faster. Reduce volume of tasks.
77. NEWRELIC
79. MONITORING IS VITAL.
80. RABBITMQ MANGEMENT PLUGIN
81. RABBITMQ MANGEMENT PLUGIN HAS A GREAT HTTP API!
82. LIBRATO Gist!
83. CELERY FLOWER
84. QUESTIONS?