Lessons learned defying Joel Spolsky with Django

Lessons learned defying Joel Spolsky with Django

Talk given at the Copenhagen Django meet up about our experiences porting Iconfinder from a LAMP stack to a de facto Python web stack.

9d198f37cdf56e90b09b5efd39b36d83?s=128

Nick Bruun

April 24, 2013
Tweet

Transcript

  1. 4.

    “… the sin le worst strate ic mistake that any

    software company can make: They decided to rewrite the code from scratch.” That’s Joel!
  2. 5.

    September 2012 LAMP stack Linux, Apache, MySQL, PHP and memcached

    (dv) (-ish) 16 cores, 16 GiB RAM, death star
  3. 9.
  4. 10.
  5. 11.
  6. 12.

    April 2013 3 M monthly visitors 3 M requests per

    day 30 GiB data in Post reSQL
  7. 13.

    April 2013 09 PM Mon 22 03 AM 06 AM

    09 AM 12 PM 03 PM 0 ms 200 ms 400 ms 600 ms 800 ms 83ms avera e response time 329ms 99th percentile response time
  8. 16.

    Watch it like a boss # In postgresql.conf: log_destination =

    'csvlog' log_directory = 'pg_log' logging_collector = on log_filename = 'postgres-%Y-%m-%d_%H%M%S' log_rotation_age = 1d log_rotation_size = 1GB log_min_duration_statement = 0ms log_checkpoints = on log_connections = on log_disconnections = on log_lock_waits = on log_temp_files = 0 dump every statement and its execution time
  9. 17.

    Watch it like a boss $ tail -f "/usr/local/var/postgres/pg_log/ $(ls

    -1t /usr/local/var/postgres/pg_log/ | head -1)" Latest log file Follow the tail of the log file
  10. 20.

    Did I say stupid? SomeModel.objects.filter( other = None ) SomeModel.objects.filter(

    other_id = None ) SELECT "myapp_somemodel"."id", "myapp_somemodel"."arbitrary", "myapp_somemodel"."other_id" FROM "myapp_somemodel" LEFT OUTER JOIN "myapp_othermodel" ON ( "myapp_somemodel"."other_id" = "myapp_othermodel"."id" ) WHERE "myapp_othermodel"."id" IS NULL SELECT "myapp_somemodel"."id", "myapp_somemodel"."arbitrary", "myapp_somemodel"."other_id" FROM "myapp_somemodel" WHERE "myapp_somemodel"."other_id" IS NULL
  11. 21.
  12. 22.

    ORMs are leaky abstractions count_by_name = {m.name: m.other_set.count() for m

    in SomeModel.objects.all()} count_by_names = {m['name']: m['num_other'] for m in SomeModel.objects.all().annotate( num_other=Count('other') ).values('name', 'num_other')} If we don’t want to waste resources like this: The essence of SQL leaks throu h:
  13. 35.

    Hot startup optimization strate y 1. Database indexes 2. Pa

    e cachin 3. Database relationship denormalization
  14. 36.

    Hot startup optimization strate y 1. Database indexes 2. Pa

    e cachin 3. Database relationship denormalization 4. Fine rained cachin
  15. 37.

    Hot startup optimization strate y 1. Database indexes 2. Pa

    e cachin 3. Database relationship denormalization 4. Fine rained cachin 5. Throw money at it
  16. 38.

    Hot startup optimization strate y 1. Database indexes 2. Pa

    e cachin 3. Database relationship denormalization 4. Fine rained cachin 5. Throw money at it 6. Go to step 5
  17. 39.
  18. 41.

    “… the sin le worst strate ic mistake that any

    software company can make: They decided to rewrite the code from scratch.” 87.91 % of time spent renderin
  19. 44.

    “… the sin le worst strate ic mistake that any

    software company can make: They decided to rewrite the code from scratch.” 29.16 % of time spent escapin HTML
  20. 54.

    Enqueuin tasks in transactions Request Back round worker Be in

    transaction Create m odel Add task to queue
  21. 55.

    Dequeue task from queue Enqueuin tasks in transactions Request Back

    round worker Be in transaction Create m odel Add task to queue
  22. 56.

    Dequeue task from queue Enqueuin tasks in transactions Request Back

    round worker Be in transaction Create m odel Add task to queue Get m odel
  23. 57.

    Dequeue task from queue Enqueuin tasks in transactions Request Back

    round worker Be in transaction Create m odel Add task to queue Get m odel raises M odel.DoesNotExist
  24. 58.

    Dequeue task from queue Enqueuin tasks in transactions Request Back

    round worker Be in transaction Create m odel Add task to queue Com m it transaction Get m odel raises M odel.DoesNotExist
  25. 63.

    With djan o-celery-transactions Request Back round worker Be in transaction

    Create m odel Locally add task to queue Com m it transaction
  26. 64.

    With djan o-celery-transactions Request Back round worker Be in transaction

    Create m odel Locally add task to queue Com m it transaction Actually add task to queue
  27. 65.

    Dequeue task from queue With djan o-celery-transactions Request Back round

    worker Be in transaction Create m odel Locally add task to queue Com m it transaction Actually add task to queue
  28. 66.

    Dequeue task from queue With djan o-celery-transactions Request Back round

    worker Be in transaction Create m odel Locally add task to queue Com m it transaction Get m odel Actually add task to queue
  29. 67.

    Dequeue task from queue With djan o-celery-transactions Request Back round

    worker Be in transaction Create m odel Locally add task to queue Com m it transaction Get m odel Actually add task to queue Finish task