Tips and Tricks from Shopify's codebase

Transcript

1. tips and tricks from Shopify’s codebase

2. None
3. None
4. None

5. measure it! if it moves...

6. statsd

7. None

8. Liquid::Template.extend StatsD::Instrument ! Liquid::Template.statsd_measure :render, 'Liquid.Template.render'

9. PaymentProcessingJob.statsd_count :perform, 'PaymentProcessingJob.processed'

10. None
11. None

12. slow queries

13. # User@Host: shopify[shopify] @ [127.0.0.1] # Thread_id: 264419969 Schema: shopify

    Last_errno: 0 Killed: 0 # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined: 147841 Rows_affected: 0 Rows_read: 147841 # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0 # InnoDB_trx_id: FF7021AAA # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: Yes Filesort_on_disk: No Merge_passes: 0 # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 475 SET timestamp=1393385020; SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` = 1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC, ends_at DESC LIMIT 1
14. None

15. # User@Host: shopify[shopify] @ [127.0.0.1] # Thread_id: 264419969 Schema: shopify

    Last_errno: 0 Killed: 0 # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined: 147841 Rows_affected: 0 Rows_read: 147841 # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0 # InnoDB_trx_id: FF7021AAA # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: Yes Filesort_on_disk: No Merge_passes: 0 # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 475 SET timestamp=1393385020; SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` = 1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC, ends_at DESC LIMIT 1 /*application:Shopify,controller:orders,action:pay*/

16. determining root cause

17. https://github.com/snormore/nginx-x-rid-header nginx request_id header proxy_set_header X-Request-ID "$request_id"; log_format main '...

    $request_id' step 1

18. https://gist.github.com/mnutt/566725 Complete 200 OK in 100ms (Views: 60ms | ActiveRecord:

    40ms | request_id=bc12813bce...) log_process_action ActionController::Instrumentation step 2

19. https://github.com/basecamp/marginalia User Load (0.3ms) SELECT `users`.* FROM `users` WHERE `users`.`id`

    = 1 LIMIT 1 /*application:Shopify, controller:users,action:show, request_id:bc12813bce...*/ basecamp/marginalia step 3

20. # User@Host: shopify[shopify] @ [127.0.0.1] # Thread_id: 264419969 Schema: shopify

    Last_errno: 0 Killed: 0 # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined: 147841 Rows_affected: 0 Rows_read: 147841 # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0 # InnoDB_trx_id: FF7021AAA # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: Yes Filesort_on_disk: No Merge_passes: 0 # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 475 SET timestamp=1393385020; SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` = 1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC, ends_at DESC LIMIT 1 /*application:Shopify,controller:orders,action:pay, request_id:bc12813bce...*/ profit!

21. access.log rails.log slow_query.log profit! (2)

22. bonus! background jobs

23. schema migration with zero downtime

24. None

25. orders 20140520_orders

26. insert/delete/update triggers

27. INSERT INTO ... SELECT ... insert/delete/update triggers

28. async caveat

29. tracking exceptions

30. None
31. None
32. None
33. None

34. thanks! :)