django in the real world (PyConPL)

Transcript

1. django in the real world yes! it scales!... YAY! Israel Fermin Montilla Software Engineer @ dubizzle August 19, 2017
2. from iferminm import more data Software Engineer @ dubizzle Venezuelan living in Dubai, UAE T: @iferminm blog: http://iffm.me
3. from dubizzle import what dubizzle is the largest classifieds website in the UAE and the region, it’s part of the OLX family.
4. from dubizzle import what dubizzle is the largest classifieds website in the UAE and the region, it’s part of the OLX family. Some other OLX family members: Avito otomoto otodom olx.pl
5. What will we see in this talk? Basic concepts The simple django project Measuring Common bottlenecks
6. Basic concepts: scalability
7. Basic concepts: scalability Scalability is the capability of a system to process or handle a growing amount of work, or its potential to be enlarged to accommodate that growth –Wikipedia
8. Basic concepts: performance
9. Basic concepts: performance Computer performance is the amount of work accomplished by a computer system. –Wikipedia
10. Basic concepts: performance Shorter response time
11. Basic concepts: performance Shorter response time
12. Basic concepts: performance Shorter response time Higher throughput (rate of processing work)
13. Basic concepts: performance Shorter response time Higher throughput (rate of processing work) Lower utilization of resources
14. Basic concepts: performance Shorter response time Higher throughput (rate of processing work) Lower utilization of resources Higher availability
15. Basic concepts: performance Shorter response time Higher throughput (rate of processing work) Lower utilization of resources Higher availability ... whatever metric you want to measure
16. Basic concepts: Pareto principle
17. Basic concepts: Pareto principle The Pareto principle states that, for many events, roughly 80% of the effects come from 20% of the causes –Wikipedia
18. Basic concepts: Pareto principle The Pareto principle states that, for many events, roughly 80% of the effects come from 20% of the causes –Wikipedia For example: 20% of the code produces 80% of the bugs.
19. Basic concepts: Pareto principle Premature optimization is bad
20. Basic concepts: Pareto principle Premature optimization is bad Optimization without measuring is bad
21. Basic concepts: Pareto principle Premature optimization is bad Optimization without measuring is bad Unprioritized optimization is bad
22. Initial django project in production Figure: Basic django project production setup
23. Profile first
24. django-debug-toolbar Figure: debug toolbar in action
25. cProfile + snakeviz Figure: snakeviz list view
26. cProfile + snakeviz Figure: snakeviz sunburst diagram
27. vprof Figure: vprof code heatmap
28. vprof Figure: vprof flame diagram
29. vprof Figure: vprof memory profiler
30. vprof Figure: vprof profiler
31. newrelic Figure: Part of newrelic’s main dashboard
32. newrelic Figure: part of newrelic’s main dashboard
33. newrelic Figure: Inside a web transaction in newrelic
34. Database
35. Reduce query counts 1 subs = S u b s c r i p t i o n . o b j e c t s . f i l t e r ( u s e r i d=u s e r . pk ) 2 f o r s i n subs : 3 packages . append ( s . package . name)
36. Reduce query counts 1 subs = S u b s c r i p t i o n . o b j e c t s . f i l t e r ( u s e r i d=u s e r . pk ) 2 f o r s i n subs : 3 packages . append ( s . package . name) N hits to the database
37. Reduce query counts 1 subs = S u b s c r i p t i o n . o b j e c t s . f i l t e r ( u s e r i d=u s e r . pk ) 2 f o r s i n subs : 3 packages . append ( s . package . name) N hits to the database 1 subs = S u b s c r i p t i o n . o b j e c t s . f i l t e r ( 2 u s e r i d=u s e r . pk 3 ) . s e l e c t r e l a t e d ( ’ package ’ )
38. Reduce query counts 1 subs = S u b s c r i p t i o n . o b j e c t s . f i l t e r ( u s e r i d=u s e r . pk ) 2 f o r s i n subs : 3 packages . append ( s . package . name) N hits to the database 1 subs = S u b s c r i p t i o n . o b j e c t s . f i l t e r ( 2 u s e r i d=u s e r . pk 3 ) . s e l e c t r e l a t e d ( ’ package ’ ) Will join the table and return it in one hit
39. Reduce query counts select related prefetch related
40. Reduce query counts Use it wisely and measure 1 u s e r = User . o b j e c t s . s e l e c t r e l a t e d ( 2 ’ sodas ’ 3 ) . get ( pk=r e q u e s t . data [ ’ u s e r i d ’ ] ) 4 5 # No a d d i t i o n a l query 6 u s e r . sodas . a l l ()
41. Reduce query counts Use it wisely and measure 1 u s e r = User . o b j e c t s . s e l e c t r e l a t e d ( 2 ’ sodas ’ 3 ) . get ( pk=r e q u e s t . data [ ’ u s e r i d ’ ] ) 4 5 # No a d d i t i o n a l query 6 u s e r . sodas . a l l () 1 # T r i g g e r s an a d d i t i o n a l query 2 u s e r . sodas . f i l t e r (name=’ p e p s i ’ ) 3 4 # Sometimes i t ’ s b e t t e r to use the cached r e s u l t 5 # and f i l t e r i n memory 6 [ s f o r s i n u s e r . sodas . a l l () i f s . name == ’ p e p s i ’ ]
42. Reduce query counts Use the Prefetch object! 1 # A product has many s u b s c r i p t i o n s and 2 # a s u b s c r i p t i o n can have many products 3 4 q u e r y s e t = S u b s c r i p t i o n . o b j e c t s . f i l t e r ( s t a t u s=e x p i r e d ) . s e l e c t r e l a t e d ( ’ c r e d i t s ’ ) 5 p r e f e t c h = P r e f e t c h ( ’ s u b s c r i p t i o n s ’ , q u e r y s e t=q u e r y s e t ) 6 7 products = Product . o b j e c t s . p r e f e t c h r e l a t e d ( p r e f e t c h ) . f i l t e r ( s e c t i o n=’ j o b s ’ )
43. Reduce query time
44. Reduce query time Indexing
45. Reduce query time Indexing 1 c l a s s U s e r P r o f i l e ( models . Model ) : 2 u s e r = models . ForeignKey ( ’ a u t h u s e r ’ ) 3 dob = models . DateField ( db index=True ) 4 e x t e r n a l i d = models . I n t e g e r F i e l d ( db index=True )
46. Reduce query time Indexing 1 c l a s s U s e r P r o f i l e ( models . Model ) : 2 u s e r = models . ForeignKey ( ’ a u t h u s e r ’ ) 3 dob = models . DateField ( db index=True ) 4 e x t e r n a l i d = models . I n t e g e r F i e l d ( db index=True ) Note: Your DBMS updates your indices in write time (INSERT and UPDATE)
47. Some notes on indexing You need to measure before you do it. Run EXPLAIN on the query (Seq scan) Index by workload If you filter on multiple columns use index together Meta option Check if the index is used before you push it. Run EXPLAIN again
48. Expensive JOINs Sometimes you might want to separate them into two different queries. 1 # You may want to see the c r e d i t spending b e h a v i o r of your u s e r s 2 C r e d i t . o b j e c t s . f i l t e r ( 3 s u b s c r i p t i o n p k g t y p e=’ motors ’ 4 ) . s e l e c t r e l a t e d ( ’ r e s o u r c e ’ ) 5 6 # Sometimes two q u e r i e s might perform b e t t e r 7 s u b s i d s = S u b s c r i p t i o n . o b j e c t s . f i l t e r ( 8 pkg type=’ motors ’ 9 ) . v a l u e s l i s t ( ’ i d ’ , f l a t=True ) 10 11 C r e d i t . o b j e c t s . f i l t e r ( 12 s u b s c r i p t i o n i d i n=s u b s i d s 13 ) . s e l e c t r e l a t e d ( ’ r e s o u r c e ’ )
49. Expensive JOINs Sometimes you might want to separate them into two different queries. 1 # You may want to see the c r e d i t spending b e h a v i o r of your u s e r s 2 C r e d i t . o b j e c t s . f i l t e r ( 3 s u b s c r i p t i o n p k g t y p e=’ motors ’ 4 ) . s e l e c t r e l a t e d ( ’ r e s o u r c e ’ ) 5 6 # Sometimes two q u e r i e s might perform b e t t e r 7 s u b s i d s = S u b s c r i p t i o n . o b j e c t s . f i l t e r ( 8 pkg type=’ motors ’ 9 ) . v a l u e s l i s t ( ’ i d ’ , f l a t=True ) 10 11 C r e d i t . o b j e c t s . f i l t e r ( 12 s u b s c r i p t i o n i d i n=s u b s i d s 13 ) . s e l e c t r e l a t e d ( ’ r e s o u r c e ’ ) ALWAYS MEASURE
50. Avoid whole table COUNT() queries After some point, having exact numbers is not important 1 PropertyForRent . o b j e c t s . count ()
51. Avoid whole table COUNT() queries After some point, having exact numbers is not important 1 PropertyForRent . o b j e c t s . count () You can instead do a raw SQL query 1 # Postgres 2 SELECT r e l t u p l e s FROM p g c l a s s 3 WHERE relname = ’ p r o p e r t y f o r r e n t ’ 4 5 # MySQL 6 SELECT t a b l e r o w s FROM information schema . t a b l e s 7 WHERE table schema = DATABASE() 8 AND table name = ’ p r o p e r t y f o r r e n t ’
52. Avoid whole table COUNT() queries After some point, having exact numbers is not important 1 PropertyForRent . o b j e c t s . count () You can instead do a raw SQL query 1 # Postgres 2 SELECT r e l t u p l e s FROM p g c l a s s 3 WHERE relname = ’ p r o p e r t y f o r r e n t ’ 4 5 # MySQL 6 SELECT t a b l e r o w s FROM information schema . t a b l e s 7 WHERE table schema = DATABASE() 8 AND table name = ’ p r o p e r t y f o r r e n t ’ This could reduce up to 90% response time
53. Use persistent connections 1 DATABASES = { 2 ’ d e f a u l t ’ : { 3 ’ENGINE ’ : ’ django . db . backends . p o s t g r e s q l p s y c o p g 2 ’ , 4 ’NAME’ : os . getenv ( ’DATABASE NAME ’ , ’ s l a y e r ’ ) , 5 ’USER ’ : os . getenv ( ’DATABASE USER ’ , None ) , 6 ’PASSWORD’ : os . getenv ( ’DATABASE PASSWORD ’ , None ) , 7 ’PORT ’ : os . getenv ( ’DATABASE PORT ’ , ’ 3306 ’ ) , 8 ’HOST ’ : os . getenv ( ’DATABASE HOST ’ , ’ l o c a l h o s t ’ ) , 9 ’CONN MAX AGE ’ : i n t ( os . getenv ( ’DATABASE CONNECTION MAX AGE ’ , ’ 0 ’ ) ) 10 } 11 }
54. Know your ORM Read the full ORM docs at least once Use F expressions to reference values within the queryset Use Q expressions for advanced filters Explore the aggregation framework Use values(), values list(), only() and defer() when the results are too big
55. Denormalize Evaluate huge joins Don’t use Generic Relations
56. Denormalize Evaluate huge joins Don’t use Generic Relations Figure: Response time reduction after denormalizing a Generic Relation
57. Query caching johny-cache django-cache-machine
58. Templates
59. Russian Doll Caching
60. Russian Doll Caching 1 {% cache MIDDLE TTL ” ads ” r e q u e s t .GET. page %} 2 {% i n c l u d e ” s e c t i o n s / p r o p e r t y / postheader . html ” %} 3 <d i v c l a s s=”ads−l i s t ”> 4 {% f o r ad i n ads %} 5 {% cache LONG TTL ” a d d e s c r i p t i o n ” a d i d ad . l a s t u p d a t e d %} 6 {% i n c l u d e ” s e c t i o n s / p r o p e r t y / a d t e a s e r . html ” %} 7 {% endcache %} 8 {% endfor %} 9 {% endcache %}
61. Further Optimization
62. Further optimization Minimize your CSS and JS (django-compressor, webassets or django-pipeline) Optimize your static images Optimize user uploaded images Serve your media and static content from a CDN Do slow work later... (celery or python-rq) Use slave replicas for read operations (and database routers)
63. Thank you! **** We’re hiring ****