Designing Fast & Scalable Python MicroServices with Django

This talk was presented at BayPIGGIES (Python User's Group in Bay Area) on Feb 22 2018. It describes a set of optimizations that one can go through in order to build fast & scalable MicroServices in Python using Django.

Transcript

1. Designing Fast & Scalable Python MicroServices with Django Dinesh A. Joshi @dineshjoshi Feb 2018
2. About • Senior Software Engineer • Distributed Systems @ Georgia Tech, Atlanta • Python Enthusiast
3. Outline • Reference App • Request & Response Pipeline • Middleware • ORM • Measuring Performance • Caching • Async Processing • Questions
4. • Python 2 or 3 • Packages ◦ django (1.9+) ◦ django-restframework ◦ django-redis ◦ psycopg2 ◦ celery • Postgres • Redis Reference App Database (Postgres) Cache (Redis) Celery Tasks Queue Django App django App
5. Request & Response Pipeline Turn a Request into a Response...
6. Request & Response Pipeline 1 WSGI Handler Middleware (Request) 2 View 3 Request duration Middleware (Response) 4 Don’t Block!
7. Middleware The code that runs in the middle...
8. • Audit & Remove unnecessary middleware • Use ◦ Conditional GET middleware ◦ GzipMiddleware ◦ Cached Sessions Middleware: Accelerating Responses
9. Middleware
10. ORM Mapping Objects to Database Table, etc...
11. • Add indexes • Enable persistent connections to database • Use Connection Pooling ORM (Settings)
12. • prefetch_related() ◦ many-to-many, many-to-one relationships • select_related() ◦ Single valued relationships • Reuse QuerySets ORM (Code)
13. len(cars) cars.count() ORM (Code)
14. ORM (Code) len(cars) cars.count()
15. Measurement Something about premature optimization being root of all evil...
16. django-debug-toolbar
17. django-debug-toolbar
18. django-debug-toolbar
19. Caching When everything else fails...
20. • Read heavy workloads • Boosts performance many-folds for cached responses • View Level or custom • Invalidation is complex Caching
21. Caching
22. Async Processing Why do it now when you can do it later…?
23. Request & Response Pipeline 1 WSGI Handler Middleware (Request) 2 View 3 Request duration Middleware (Response) 4 Don’t Block!
24. Async Processing Database (Postgres) Cache (Redis) Celery Tasks Queue Django App django App 1 2 3 Service X 4
25. Questions?
26. Contact Dinesh Joshi dinesh.joshi@gatech.edu www.dineshjoshi.com @dineshjoshi