Lessons learned defying Joel Spolsky with Django

Talk given at the Copenhagen Django meet up about our experiences porting Iconfinder from a LAMP stack to a de facto Python web stack.

Transcript

1. Stayin sane while defyin Joel Spolsky with Djan o
2. Hola, me llamo Nick nickbruun @nickbruun http://bruun.co/
3. I’m the CTO of Iconfinder
4. “… the sin le worst strate ic mistake that any software company can make: They decided to rewrite the code from scratch.” That’s Joel!
5. September 2012 LAMP stack Linux, Apache, MySQL, PHP and memcached (dv) (-ish) 16 cores, 16 GiB RAM, death star
6. Scientific chart required processin power Not so far future Sep 2012 May 2007 0 limit of (dv)
7. Restructure Rebuild
8. Fast and flexible Two simple oals: (aka code base nirvana)
12. April 2013 3 M monthly visitors 3 M requests per day 30 GiB data in Post reSQL
13. April 2013 09 PM Mon 22 03 AM 06 AM 09 AM 12 PM 03 PM 0 ms 200 ms 400 ms 600 ms 800 ms 83ms avera e response time 329ms 99th percentile response time
14. We survived! Lon story short… - and learned a few lessons in the process
15. 1. ORMs are stupid.
16. Watch it like a boss # In postgresql.conf: log_destination = 'csvlog' log_directory = 'pg_log' logging_collector = on log_filename = 'postgres-%Y-%m-%d_%H%M%S' log_rotation_age = 1d log_rotation_size = 1GB log_min_duration_statement = 0ms log_checkpoints = on log_connections = on log_disconnections = on log_lock_waits = on log_temp_files = 0 dump every statement and its execution time
17. Watch it like a boss $ tail -f "/usr/local/var/postgres/pg_log/ $(ls -1t /usr/local/var/postgres/pg_log/ | head -1)" Latest log file Follow the tail of the log file
18. Did I say stupid? SomeModel.objects.filter( other = None ) SomeModel.objects.filter( other_id = None )
19. Did I say stupid? SomeModel.objects.filter( other = None ) SomeModel.objects.filter( other_id = None ) same exact thing!
20. Did I say stupid? SomeModel.objects.filter( other = None ) SomeModel.objects.filter( other_id = None ) SELECT "myapp_somemodel"."id", "myapp_somemodel"."arbitrary", "myapp_somemodel"."other_id" FROM "myapp_somemodel" LEFT OUTER JOIN "myapp_othermodel" ON ( "myapp_somemodel"."other_id" = "myapp_othermodel"."id" ) WHERE "myapp_othermodel"."id" IS NULL SELECT "myapp_somemodel"."id", "myapp_somemodel"."arbitrary", "myapp_somemodel"."other_id" FROM "myapp_somemodel" WHERE "myapp_somemodel"."other_id" IS NULL
22. ORMs are leaky abstractions count_by_name = {m.name: m.other_set.count() for m in SomeModel.objects.all()} count_by_names = {m['name']: m['num_other'] for m in SomeModel.objects.all().annotate( num_other=Count('other') ).values('name', 'num_other')} If we don’t want to waste resources like this: The essence of SQL leaks throu h:
23. I, for one, welcome our SQL overlords
24. 2. Monitor everythin .
25. Graphite Sentry
26. No, your lo file probably isn’t ood enou h.
27. build stuff Side effect: awesome deployment workflow
28. build stuff deploy Side effect: awesome deployment workflow
29. build stuff deploy watch exceptions Side effect: awesome deployment workflow
30. build stuff deploy watch exceptions Side effect: awesome deployment workflow
31. 3. Don’t assume. Profile.
32. Hot startup optimization strate y
33. Hot startup optimization strate y 1. Database indexes
34. Hot startup optimization strate y 1. Database indexes 2. Pa e cachin
35. Hot startup optimization strate y 1. Database indexes 2. Pa e cachin 3. Database relationship denormalization
36. Hot startup optimization strate y 1. Database indexes 2. Pa e cachin 3. Database relationship denormalization 4. Fine rained cachin
37. Hot startup optimization strate y 1. Database indexes 2. Pa e cachin 3. Database relationship denormalization 4. Fine rained cachin 5. Throw money at it
38. Hot startup optimization strate y 1. Database indexes 2. Pa e cachin 3. Database relationship denormalization 4. Fine rained cachin 5. Throw money at it 6. Go to step 5
40. 91ms ori inal response time
41. “… the sin le worst strate ic mistake that any software company can make: They decided to rewrite the code from scratch.” 87.91 % of time spent renderin
42. Japanese temple
43. 29ms response time with Jinja2 91ms ori inal response time
44. “… the sin le worst strate ic mistake that any software company can make: They decided to rewrite the code from scratch.” 29.16 % of time spent escapin HTML
45. 20ms response time with Jinja2 without auto-escapin 29ms response time with Jinja2 91ms ori inal response time
46. 4. Everythin prepacka ed isn’t ood, and everythin ood isn’t prepacka ed.
47. Watch your transactions Transactions
48. Watch your transactions Transactions Cachin +
49. Watch your transactions Transactions Back round tasks + Cachin +
50. Watch your transactions Transactions = ANGER!!1 Back round tasks + Cachin +
51. Enqueuin tasks in transactions Request Back round worker
52. Enqueuin tasks in transactions Request Back round worker Be in transaction
53. Enqueuin tasks in transactions Request Back round worker Be in transaction Create m odel
54. Enqueuin tasks in transactions Request Back round worker Be in transaction Create m odel Add task to queue
55. Dequeue task from queue Enqueuin tasks in transactions Request Back round worker Be in transaction Create m odel Add task to queue
56. Dequeue task from queue Enqueuin tasks in transactions Request Back round worker Be in transaction Create m odel Add task to queue Get m odel
57. Dequeue task from queue Enqueuin tasks in transactions Request Back round worker Be in transaction Create m odel Add task to queue Get m odel raises M odel.DoesNotExist
58. Dequeue task from queue Enqueuin tasks in transactions Request Back round worker Be in transaction Create m odel Add task to queue Com m it transaction Get m odel raises M odel.DoesNotExist
59. With djan o-celery-transactions Request Back round worker
60. With djan o-celery-transactions Request Back round worker Be in transaction
61. With djan o-celery-transactions Request Back round worker Be in transaction Create m odel
62. With djan o-celery-transactions Request Back round worker Be in transaction Create m odel Locally add task to queue
63. With djan o-celery-transactions Request Back round worker Be in transaction Create m odel Locally add task to queue Com m it transaction
64. With djan o-celery-transactions Request Back round worker Be in transaction Create m odel Locally add task to queue Com m it transaction Actually add task to queue
65. Dequeue task from queue With djan o-celery-transactions Request Back round worker Be in transaction Create m odel Locally add task to queue Com m it transaction Actually add task to queue
66. Dequeue task from queue With djan o-celery-transactions Request Back round worker Be in transaction Create m odel Locally add task to queue Com m it transaction Get m odel Actually add task to queue
67. Dequeue task from queue With djan o-celery-transactions Request Back round worker Be in transaction Create m odel Locally add task to queue Com m it transaction Get m odel Actually add task to queue Finish task
68. is awesome
69. is a trap!
70. are a trap! Frameworks
71. Thanks! It’s been awesome listenin to me.