Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Tips and Tricks from Shopify's codebase

Tips and Tricks from Shopify's codebase

Talk given at Montreal.rb on May 20th, 2014

Christian Joudrey

May 23, 2014
Tweet

More Decks by Christian Joudrey

Other Decks in Programming

Transcript

  1. tips and tricks
    from Shopify’s codebase

    View Slide

  2. View Slide

  3. View Slide

  4. View Slide

  5. measure it!
    if it moves...

    View Slide

  6. statsd

    View Slide

  7. View Slide

  8. Liquid::Template.extend StatsD::Instrument
    !
    Liquid::Template.statsd_measure :render,
    'Liquid.Template.render'

    View Slide

  9. PaymentProcessingJob.statsd_count :perform,
    'PaymentProcessingJob.processed'

    View Slide

  10. View Slide

  11. View Slide

  12. slow queries

    View Slide

  13. # [email protected]: shopify[shopify] @ [127.0.0.1]
    # Thread_id: 264419969 Schema: shopify Last_errno: 0 Killed: 0
    # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined:
    147841 Rows_affected: 0 Rows_read: 147841
    # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0
    # InnoDB_trx_id: FF7021AAA
    # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No
    Tmp_table_on_disk: No
    # Filesort: Yes Filesort_on_disk: No Merge_passes: 0
    # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000
    # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000
    # InnoDB_pages_distinct: 475
    SET timestamp=1393385020;
    SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` =
    1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC,
    ends_at DESC LIMIT 1

    View Slide

  14. View Slide

  15. # [email protected]: shopify[shopify] @ [127.0.0.1]
    # Thread_id: 264419969 Schema: shopify Last_errno: 0 Killed: 0
    # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined:
    147841 Rows_affected: 0 Rows_read: 147841
    # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0
    # InnoDB_trx_id: FF7021AAA
    # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No
    Tmp_table_on_disk: No
    # Filesort: Yes Filesort_on_disk: No Merge_passes: 0
    # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000
    # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000
    # InnoDB_pages_distinct: 475
    SET timestamp=1393385020;
    SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` =
    1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC,
    ends_at DESC LIMIT 1 /*application:Shopify,controller:orders,action:pay*/

    View Slide

  16. determining
    root cause

    View Slide

  17. https://github.com/snormore/nginx-x-rid-header
    nginx request_id header
    proxy_set_header X-Request-ID "$request_id";
    log_format main '... $request_id'
    step 1

    View Slide

  18. https://gist.github.com/mnutt/566725
    Complete 200 OK in 100ms (Views: 60ms |
    ActiveRecord: 40ms | request_id=bc12813bce...)
    log_process_action
    ActionController::Instrumentation
    step 2

    View Slide

  19. https://github.com/basecamp/marginalia
    User Load (0.3ms) SELECT `users`.* FROM `users`
    WHERE `users`.`id` = 1 LIMIT 1
    /*application:Shopify,
    controller:users,action:show,
    request_id:bc12813bce...*/
    basecamp/marginalia
    step 3

    View Slide

  20. # [email protected]: shopify[shopify] @ [127.0.0.1]
    # Thread_id: 264419969 Schema: shopify Last_errno: 0 Killed: 0
    # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined:
    147841 Rows_affected: 0 Rows_read: 147841
    # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0
    # InnoDB_trx_id: FF7021AAA
    # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No
    Tmp_table_on_disk: No
    # Filesort: Yes Filesort_on_disk: No Merge_passes: 0
    # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000
    # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000
    # InnoDB_pages_distinct: 475
    SET timestamp=1393385020;
    SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` =
    1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC,
    ends_at DESC LIMIT 1 /*application:Shopify,controller:orders,action:pay,
    request_id:bc12813bce...*/
    profit!

    View Slide

  21. access.log
    rails.log
    slow_query.log
    profit! (2)

    View Slide

  22. bonus!
    background jobs

    View Slide

  23. schema migration
    with zero downtime

    View Slide

  24. View Slide

  25. orders 20140520_orders

    View Slide

  26. insert/delete/update triggers

    View Slide

  27. INSERT INTO ... SELECT ...
    insert/delete/update triggers

    View Slide

  28. async
    caveat

    View Slide

  29. tracking exceptions

    View Slide

  30. View Slide

  31. View Slide

  32. View Slide

  33. View Slide

  34. thanks! :)

    View Slide