Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Tips and Tricks from Shopify's codebase

Tips and Tricks from Shopify's codebase

Talk given at Montreal.rb on May 20th, 2014

Christian Joudrey

May 23, 2014
Tweet

More Decks by Christian Joudrey

Other Decks in Programming

Transcript

  1. tips and tricks
    from Shopify’s codebase

    View full-size slide

  2. measure it!
    if it moves...

    View full-size slide

  3. Liquid::Template.extend StatsD::Instrument
    !
    Liquid::Template.statsd_measure :render,
    'Liquid.Template.render'

    View full-size slide

  4. PaymentProcessingJob.statsd_count :perform,
    'PaymentProcessingJob.processed'

    View full-size slide

  5. slow queries

    View full-size slide

  6. # User@Host: shopify[shopify] @ [127.0.0.1]
    # Thread_id: 264419969 Schema: shopify Last_errno: 0 Killed: 0
    # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined:
    147841 Rows_affected: 0 Rows_read: 147841
    # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0
    # InnoDB_trx_id: FF7021AAA
    # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No
    Tmp_table_on_disk: No
    # Filesort: Yes Filesort_on_disk: No Merge_passes: 0
    # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000
    # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000
    # InnoDB_pages_distinct: 475
    SET timestamp=1393385020;
    SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` =
    1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC,
    ends_at DESC LIMIT 1

    View full-size slide

  7. # User@Host: shopify[shopify] @ [127.0.0.1]
    # Thread_id: 264419969 Schema: shopify Last_errno: 0 Killed: 0
    # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined:
    147841 Rows_affected: 0 Rows_read: 147841
    # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0
    # InnoDB_trx_id: FF7021AAA
    # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No
    Tmp_table_on_disk: No
    # Filesort: Yes Filesort_on_disk: No Merge_passes: 0
    # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000
    # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000
    # InnoDB_pages_distinct: 475
    SET timestamp=1393385020;
    SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` =
    1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC,
    ends_at DESC LIMIT 1 /*application:Shopify,controller:orders,action:pay*/

    View full-size slide

  8. determining
    root cause

    View full-size slide

  9. https://github.com/snormore/nginx-x-rid-header
    nginx request_id header
    proxy_set_header X-Request-ID "$request_id";
    log_format main '... $request_id'
    step 1

    View full-size slide

  10. https://gist.github.com/mnutt/566725
    Complete 200 OK in 100ms (Views: 60ms |
    ActiveRecord: 40ms | request_id=bc12813bce...)
    log_process_action
    ActionController::Instrumentation
    step 2

    View full-size slide

  11. https://github.com/basecamp/marginalia
    User Load (0.3ms) SELECT `users`.* FROM `users`
    WHERE `users`.`id` = 1 LIMIT 1
    /*application:Shopify,
    controller:users,action:show,
    request_id:bc12813bce...*/
    basecamp/marginalia
    step 3

    View full-size slide

  12. # User@Host: shopify[shopify] @ [127.0.0.1]
    # Thread_id: 264419969 Schema: shopify Last_errno: 0 Killed: 0
    # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined:
    147841 Rows_affected: 0 Rows_read: 147841
    # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0
    # InnoDB_trx_id: FF7021AAA
    # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No
    Tmp_table_on_disk: No
    # Filesort: Yes Filesort_on_disk: No Merge_passes: 0
    # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000
    # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000
    # InnoDB_pages_distinct: 475
    SET timestamp=1393385020;
    SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` =
    1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC,
    ends_at DESC LIMIT 1 /*application:Shopify,controller:orders,action:pay,
    request_id:bc12813bce...*/
    profit!

    View full-size slide

  13. access.log
    rails.log
    slow_query.log
    profit! (2)

    View full-size slide

  14. bonus!
    background jobs

    View full-size slide

  15. schema migration
    with zero downtime

    View full-size slide

  16. orders 20140520_orders

    View full-size slide

  17. insert/delete/update triggers

    View full-size slide

  18. INSERT INTO ... SELECT ...
    insert/delete/update triggers

    View full-size slide

  19. tracking exceptions

    View full-size slide