Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Lessons learned defying Joel Spolsky with Django

Lessons learned defying Joel Spolsky with Django

Talk given at the Copenhagen Django meet up about our experiences porting Iconfinder from a LAMP stack to a de facto Python web stack.

Nick Bruun

April 24, 2013
Tweet

More Decks by Nick Bruun

Other Decks in Programming

Transcript

  1. Stayin sane while defyin
    Joel Spolsky with Djan o

    View Slide

  2. Hola, me llamo Nick
    nickbruun
    @nickbruun http://bruun.co/

    View Slide

  3. I’m the CTO
    of Iconfinder

    View Slide

  4. “… the sin le worst strate ic
    mistake that any software
    company can make: They decided
    to rewrite the code from scratch.”
    That’s Joel!

    View Slide

  5. September 2012
    LAMP stack
    Linux, Apache, MySQL, PHP
    and memcached
    (dv) (-ish)
    16 cores, 16 GiB RAM,
    death star

    View Slide

  6. Scientific chart
    required processin power
    Not so far future
    Sep 2012
    May 2007
    0
    limit of (dv)

    View Slide

  7. Restructure
    Rebuild

    View Slide

  8. Fast and flexible
    Two simple oals:
    (aka code base nirvana)

    View Slide

  9. View Slide

  10. View Slide

  11. View Slide

  12. April 2013
    3
    M
    monthly visitors
    3
    M
    requests per day
    30
    GiB
    data in Post reSQL

    View Slide

  13. April 2013
    09 PM Mon 22 03 AM 06 AM 09 AM 12 PM 03 PM
    0 ms
    200 ms
    400 ms
    600 ms
    800 ms
    83ms
    avera e response time
    329ms
    99th percentile response time

    View Slide

  14. We survived!
    Lon story short…
    - and learned a few lessons
    in the process

    View Slide

  15. 1. ORMs are stupid.

    View Slide

  16. Watch it like a boss
    # In postgresql.conf:
    log_destination = 'csvlog'
    log_directory = 'pg_log'
    logging_collector = on
    log_filename = 'postgres-%Y-%m-%d_%H%M%S'
    log_rotation_age = 1d
    log_rotation_size = 1GB
    log_min_duration_statement = 0ms
    log_checkpoints = on
    log_connections = on
    log_disconnections = on
    log_lock_waits = on
    log_temp_files = 0
    dump every statement
    and its execution time

    View Slide

  17. Watch it like a boss
    $ tail -f "/usr/local/var/postgres/pg_log/
    $(ls -1t /usr/local/var/postgres/pg_log/ | head -1)"
    Latest log file
    Follow the tail of
    the log file

    View Slide

  18. Did I say stupid?
    SomeModel.objects.filter(
    other = None
    )
    SomeModel.objects.filter(
    other_id = None
    )

    View Slide

  19. Did I say stupid?
    SomeModel.objects.filter(
    other = None
    )
    SomeModel.objects.filter(
    other_id = None
    )
    same exact thing!

    View Slide

  20. Did I say stupid?
    SomeModel.objects.filter(
    other = None
    )
    SomeModel.objects.filter(
    other_id = None
    )
    SELECT
    "myapp_somemodel"."id",
    "myapp_somemodel"."arbitrary",
    "myapp_somemodel"."other_id"
    FROM
    "myapp_somemodel"
    LEFT OUTER JOIN "myapp_othermodel" ON (
    "myapp_somemodel"."other_id" =
    "myapp_othermodel"."id"
    )
    WHERE
    "myapp_othermodel"."id" IS NULL
    SELECT
    "myapp_somemodel"."id",
    "myapp_somemodel"."arbitrary",
    "myapp_somemodel"."other_id"
    FROM
    "myapp_somemodel"
    WHERE
    "myapp_somemodel"."other_id" IS NULL

    View Slide

  21. View Slide

  22. ORMs are leaky abstractions
    count_by_name = {m.name: m.other_set.count() for m in SomeModel.objects.all()}
    count_by_names = {m['name']: m['num_other'] for m in SomeModel.objects.all().annotate(
    num_other=Count('other')
    ).values('name', 'num_other')}
    If we don’t want to waste resources like this:
    The essence of SQL leaks throu h:

    View Slide

  23. I, for one, welcome our
    SQL overlords

    View Slide

  24. 2. Monitor everythin .

    View Slide

  25. Graphite
    Sentry

    View Slide

  26. No, your lo file probably isn’t ood enou h.

    View Slide

  27. build stuff
    Side effect:
    awesome deployment workflow

    View Slide

  28. build stuff deploy
    Side effect:
    awesome deployment workflow

    View Slide

  29. build stuff deploy watch exceptions
    Side effect:
    awesome deployment workflow

    View Slide

  30. build stuff deploy watch exceptions
    Side effect:
    awesome deployment workflow

    View Slide

  31. 3. Don’t assume. Profile.

    View Slide

  32. Hot startup optimization strate y

    View Slide

  33. Hot startup optimization strate y
    1. Database indexes

    View Slide

  34. Hot startup optimization strate y
    1. Database indexes
    2. Pa e cachin

    View Slide

  35. Hot startup optimization strate y
    1. Database indexes
    2. Pa e cachin
    3. Database relationship denormalization

    View Slide

  36. Hot startup optimization strate y
    1. Database indexes
    2. Pa e cachin
    3. Database relationship denormalization
    4. Fine rained cachin

    View Slide

  37. Hot startup optimization strate y
    1. Database indexes
    2. Pa e cachin
    3. Database relationship denormalization
    4. Fine rained cachin
    5. Throw money at it

    View Slide

  38. Hot startup optimization strate y
    1. Database indexes
    2. Pa e cachin
    3. Database relationship denormalization
    4. Fine rained cachin
    5. Throw money at it
    6. Go to step 5

    View Slide

  39. View Slide

  40. 91ms
    ori inal
    response time

    View Slide

  41. “… the sin le worst strate ic
    mistake that any software
    company can make: They decided
    to rewrite the code from scratch.”
    87.91 % of time spent renderin

    View Slide

  42. Japanese temple

    View Slide

  43. 29ms
    response time
    with Jinja2
    91ms
    ori inal
    response time

    View Slide

  44. “… the sin le worst strate ic
    mistake that any software
    company can make: They decided
    to rewrite the code from scratch.”
    29.16 % of time spent escapin HTML

    View Slide

  45. 20ms
    response time with
    Jinja2 without auto-escapin
    29ms
    response time
    with Jinja2
    91ms
    ori inal
    response time

    View Slide

  46. 4. Everythin prepacka ed isn’t
    ood, and everythin ood isn’t
    prepacka ed.

    View Slide

  47. Watch your transactions
    Transactions

    View Slide

  48. Watch your transactions
    Transactions Cachin
    +

    View Slide

  49. Watch your transactions
    Transactions Back round
    tasks
    +
    Cachin
    +

    View Slide

  50. Watch your transactions
    Transactions
    =
    ANGER!!1
    Back round
    tasks
    +
    Cachin
    +

    View Slide

  51. Enqueuin tasks in transactions
    Request
    Back round worker

    View Slide

  52. Enqueuin tasks in transactions
    Request
    Back round worker
    Be
    in
    transaction

    View Slide

  53. Enqueuin tasks in transactions
    Request
    Back round worker
    Be
    in
    transaction
    Create m
    odel

    View Slide

  54. Enqueuin tasks in transactions
    Request
    Back round worker
    Be
    in
    transaction
    Create m
    odel
    Add
    task to
    queue

    View Slide

  55. Dequeue task from
    queue
    Enqueuin tasks in transactions
    Request
    Back round worker
    Be
    in
    transaction
    Create m
    odel
    Add
    task to
    queue

    View Slide

  56. Dequeue task from
    queue
    Enqueuin tasks in transactions
    Request
    Back round worker
    Be
    in
    transaction
    Create m
    odel
    Add
    task to
    queue
    Get m
    odel

    View Slide

  57. Dequeue task from
    queue
    Enqueuin tasks in transactions
    Request
    Back round worker
    Be
    in
    transaction
    Create m
    odel
    Add
    task to
    queue
    Get m
    odel
    raises M
    odel.DoesNotExist

    View Slide

  58. Dequeue task from
    queue
    Enqueuin tasks in transactions
    Request
    Back round worker
    Be
    in
    transaction
    Create m
    odel
    Add
    task to
    queue
    Com
    m
    it transaction
    Get m
    odel
    raises M
    odel.DoesNotExist

    View Slide

  59. With djan o-celery-transactions
    Request
    Back round worker

    View Slide

  60. With djan o-celery-transactions
    Request
    Back round worker
    Be
    in
    transaction

    View Slide

  61. With djan o-celery-transactions
    Request
    Back round worker
    Be
    in
    transaction
    Create m
    odel

    View Slide

  62. With djan o-celery-transactions
    Request
    Back round worker
    Be
    in
    transaction
    Create m
    odel
    Locally add
    task to
    queue

    View Slide

  63. With djan o-celery-transactions
    Request
    Back round worker
    Be
    in
    transaction
    Create m
    odel
    Locally add
    task to
    queue
    Com
    m
    it transaction

    View Slide

  64. With djan o-celery-transactions
    Request
    Back round worker
    Be
    in
    transaction
    Create m
    odel
    Locally add
    task to
    queue
    Com
    m
    it transaction
    Actually add
    task to
    queue

    View Slide

  65. Dequeue task from
    queue
    With djan o-celery-transactions
    Request
    Back round worker
    Be
    in
    transaction
    Create m
    odel
    Locally add
    task to
    queue
    Com
    m
    it transaction
    Actually add
    task to
    queue

    View Slide

  66. Dequeue task from
    queue
    With djan o-celery-transactions
    Request
    Back round worker
    Be
    in
    transaction
    Create m
    odel
    Locally add
    task to
    queue
    Com
    m
    it transaction
    Get m
    odel
    Actually add
    task to
    queue

    View Slide

  67. Dequeue task from
    queue
    With djan o-celery-transactions
    Request
    Back round worker
    Be
    in
    transaction
    Create m
    odel
    Locally add
    task to
    queue
    Com
    m
    it transaction
    Get m
    odel
    Actually add
    task to
    queue
    Finish
    task

    View Slide

  68. is awesome

    View Slide

  69. is a trap!

    View Slide

  70. are a trap!
    Frameworks

    View Slide

  71. Thanks!
    It’s been awesome listenin to me.

    View Slide