Async Processing with Django and Celery

Presentation given at the Northern Colorado Python Web meetup on August 11, 2014

Transcript

1. Asynchronous Processing with Celery and Django Presented at Fort Collins Python Web meetup August 11, 2014 1 / 23
2. Who Am I? Senior Web Application Developer at Kansas State University (Full time Remote) Working with Django/Python for 4+ years Twitter: @derekstegelman GitHub: @dstegelman 2 / 23
3. What is it and why should I be using it? 3 / 23
4. In computer programming, asynchronous events are those occurring independently of the main program flow. Asynchronous actions are actions executed in a non-blocking scheme, allowing the main program flow to continue processing. 4 / 23
5. Differences Synchronous d e f b l o c k i n g _ v i e w ( r e q u e s t , o b j e c t _ p k = N o n e ) : o b j e c t = g e t _ o b j e c t _ o r _ 4 0 4 ( S o m e O b j e c t , p k = o b j e c t _ p k ) o b j e c t . d o _ m e t h o d ( ) # B l o c k i n g . W e c a n ' t s h o w t h e t e m p l a t e u n t i l t h i s c o m p l e t e s o b j e c t . d o _ l o n g _ t r a n s a c t i o n ( ) r e t u r n r e n d e r ( r e q u e s t , ' t e m p l a t e . h t m l ' ) Asynchronous d e f b l o c k i n g _ v i e w ( r e q u e s t , o b j e c t _ p k = N o n e ) : o b j e c t = g e t _ o b j e c t _ o r _ 4 0 4 ( S o m e O b j e c t , p k = o b j e c t _ p k ) o b j e c t . d o _ m e t h o d ( ) # N o n - b l o c k i n g . C e l e r y t a k e s o v e r t h i s c o d e a n d l e t s u s m o v e r i g h t a l o n g O b j e c t T r a n s a c t i o n . d e l a y ( o b j e c t ) r e t u r n r e n d e r ( r e q u e s t , ' t e m p l a t e . h t m l ' ) 5 / 23
6. Asynchronous processing is an easy way to defer processing of non-real time dependent code. f r o m c e l e r y . t a s k i m p o r t t a s k @ t a s k ( n a m e = ' p h o t o _ m a n a g e r . c l e a r _ t h u m b n a i l s ' ) d e f c l e a r _ t h u m b n a i l s ( p h o t o ) : " " " @ t o d o - A d d C o m m e n t s " " " p h o t o . _ c l e a r _ t h u m b n a i l s ( ) @ t a s k ( n a m e = ' p h o t o _ m a n a g e r . b u i l d _ t h u m b n a i l s ' ) d e f b u i l d _ t h u m b n a i l s ( p h o t o ) : " " " @ t o d o - A d d C o m m e n t s " " " p h o t o . m a k e _ t h u m b n a i l s ( ) If some code can be taken out of the request/response cycle it should. 6 / 23
7. What is it? Celery is a robust task worker solution implemented in Python that can consume tasks from numerous types of message brokers. Some highlights include: Good support for both RabbitMQ and Redis as message brokers. (Others can be used but not officially supported.) Easily scalable, supports autoscaling, and clustering is fairly easy. Support for scheduled and delayed tasks Can be monitored using Celery Flower 7 / 23
8. What is it? Why use it? Asynchronous processing can be a great solution for several different types of issues. Long running processes that do not need to return a response to the browser Scheduled tasks that would otherwise be performed with cron Clean up tasks (Auto clean up bad user records, etc.) 8 / 23
9. What is it? Why use it? Use Cases Any work that can doesn't need to return a response to the user is a good candidate for async processing. Some use cases include: Sending Emails Additional processing on transactions Sending anyaltyics Processing thumbnails for photos Cleaning and sanitizing data 9 / 23
10. Installation and integration with Django 10 / 23
11. Quick and easy install with Django Celery can of course be used outside of Django Install Redis or RabbitMQ for your distro p i p i n s t a l l c e l e r y Configure Celery.py f r o m _ _ f u t u r e _ _ i m p o r t a b s o l u t e _ i m p o r t i m p o r t o s f r o m c e l e r y i m p o r t C e l e r y f r o m d j a n g o . c o n f i m p o r t s e t t i n g s # s e t t h e d e f a u l t D j a n g o s e t t i n g s m o d u l e f o r t h e ' c e l e r y ' p r o g r a m . o s . e n v i r o n . s e t d e f a u l t ( ' D J A N G O _ S E T T I N G S _ M O D U L E ' , ' c e l e r y _ d e m o . s e t t i n g s ' ) a p p = C e l e r y ( ' c e l e r y _ d e m o ' ) # U s i n g a s t r i n g h e r e m e a n s t h e w o r k e r w i l l n o t h a v e t o # p i c k l e t h e o b j e c t w h e n u s i n g W i n d o w s . a p p . c o n f i g _ f r o m _ o b j e c t ( ' d j a n g o . c o n f : s e t t i n g s ' ) a p p . a u t o d i s c o v e r _ t a s k s ( l a m b d a : s e t t i n g s . I N S T A L L E D _ A P P S ) 11 / 23
12. Quick and easy install with Django - Settings.py f r o m _ _ f u t u r e _ _ i m p o r t a b s o l u t e _ i m p o r t # ^ ^ ^ T h e a b o v e i s r e q u i r e d i f y o u w a n t t o i m p o r t f r o m t h e c e l e r y # l i b r a r y . I f y o u d o n ' t h a v e t h i s t h e n ` f r o m c e l e r y . s c h e d u l e s i m p o r t ` # b e c o m e s ` p r o j . c e l e r y . s c h e d u l e s ` i n P y t h o n 2 . x s i n c e i t a l l o w s # f o r r e l a t i v e i m p o r t s b y d e f a u l t . # C e l e r y s e t t i n g s B R O K E R _ U R L = ' a m q p : / / g u e s t : g u e s t @ l o c a l h o s t / ' # : O n l y a d d p i c k l e t o t h i s l i s t i f y o u r b r o k e r i s s e c u r e d # : f r o m u n w a n t e d a c c e s s ( s e e u s e r g u i d e / s e c u r i t y . h t m l ) C E L E R Y _ A C C E P T _ C O N T E N T = [ ' j s o n ' ] C E L E R Y _ T A S K _ S E R I A L I Z E R = ' j s o n ' C E L E R Y _ R E S U L T _ S E R I A L I Z E R = ' j s o n ' C E L E R Y B E A T _ S C H E D U L E R = ' d j c e l e r y . s c h e d u l e r s . D a t a b a s e S c h e d u l e r ' 12 / 23
13. A Basic Celery Task 13 / 23
14. Task Creation Celery tasks are just python methods that either inherit from the Task class or are decorated with @ s h a r e d _ t a s k f r o m _ _ f u t u r e _ _ i m p o r t a b s o l u t e _ i m p o r t i m p o r t t i m e f r o m c e l e r y i m p o r t s h a r e d _ t a s k @ s h a r e d _ t a s k d e f a d d ( x , y ) : p r i n t ( x + y ) r e t u r n x + y Tasks are then called in your code with the . d e l a y ( ) method. f r o m m y m o d u l e . t a s k s i m p o r t a d d a d d . d e l a y ( 3 , 8 ) 14 / 23
15. Demo 15 / 23
16. Deployment with Supervisor 16 / 23
17. Deployment Deployment is straightforward. The typical way to start and stop Celery in production is with Supervisor. [ p r o g r a m : d e r e k _ s t e g e l m a n _ c o m _ c e l e r y ] c o m m a n d = / s r v / w w w / v i r t u a l e n v s / d e r e k - s t e g e l m a n - c o m / b i n / p y t h o n / s r v / w w w / p r o j e c t s / d e r e k - s t e g e l m a n - c o d i r e c t o r y = / s r v / w w w / p r o j e c t s / d e r e k - s t e g e l m a n - c o m u s e r = d s t e g e l m a n a u t o s t a r t = t r u e a u t o r e s t a r t = t r u e r e d i r e c t _ s t d e r r = T r u e Celery must be restarted after new code is deployed!! 17 / 23
18. Monitoring Celery with Flower 18 / 23
19. Monitor with Flower 19 / 23
20. Demo 20 / 23
21. Debugging Propogate errors to something like Sentry. Good idea to tail the log right after deployments for heavy use sites Add an assert false to make sure errors are propagated in your configuration. 21 / 23
22. Next Steps/Further Reading We just scratched the surface of what you can accomplish using Celery. Take a look at the docs for further information. Celery Docs Celery Flower Django Docs Supervisor This Presentation/Demo Code 22 / 23
23. Thanks/Questions?! Twitter: @derekstegelman Github: dstegelman 23 / 23