Scaling Shopify

Talk given at ConFoo 2014 on February 28th, 2014.

Transcript

1. SCALING SHOPIFY
   ...or ensuring happiness for online shoppers
   

   View Slide

2. cjoudrey  
   @
   

   View Slide

3. View Slide

4. View Slide

5. View Slide

6. View Slide

7. the stack
   

   View Slide

8. nginx unicorn
   • rails 4
   •
   mysql 5.6 (percona)
   ruby 2.1 •
   

   View Slide

9. 95 app servers 3,884 unicorn workers
   5 job servers 387 job workers
   

   View Slide

10. 1 request 1 process
    =
    

    View Slide

11. scale?
    

    View Slide

12. over 90,000 shops
    

    View Slide

13. View Slide

14. 1.6B$ annual GMV
    that’s 3,600$ per min
    

    View Slide

15. cyber monday
    black friday
    

    View Slide

16. View Slide

17. 61 M$ in GMV
    in four days
    

    View Slide

18. flash sales
    

    View Slide

19. View Slide

20. View Slide

21. page caching
    

    View Slide

22. View Slide

23. View Slide

24. shopify/cacheable
    

    View Slide

25. generational caching
    

    View Slide

26. gzip • etag + 304 not modified
    

    View Slide

27. class PostsController < ApplicationController
    def show
    response_cache do
    @post = @shop.posts.find(params[:id])
    respond_with(@post)
    end
    end
    def cache_key_data
    {
    action: action_name,
    format: request.format,
    params: params.slice(:id),
    shop_version: @shop.version
    }
    end
    end
    

    View Slide

28. View Slide

29. flash sale
    

    View Slide

30. query caching
    

    View Slide

31. shopify/identity_cache
    

    View Slide

32. full model caching
    

    View Slide

33. opt-in by design
    

    View Slide

34. after_commit expiry
    

    View Slide

35. class Product < ActiveRecord::Base
    include IdentityCache
    has_many :images
    cache_has_many :images, :embed => true
    end
    product = Product.fetch(id)
    images = product.fetch_images
    

    View Slide

36. class Product < ActiveRecord::Base
    include IdentityCache
    cache_index :shop_id, :handle, :unique => true
    end
    Product.fetch_by_shop_id_and_handle(shop_id, handle)
    

    View Slide

37. View Slide

38. flash sale
    

    View Slide

39. background jobs
    

    View Slide

40. webhooks emails
    • fraud detection
    •
    payment processing
    

    View Slide

41. View Slide

42. priority queues
    payment
    •
    default •
    low realtime
    •
    

    View Slide

43. class ProductImportJob
    include BackgroundQueue::Realtime
    def perform(params)
    ...
    end
    end
    BackgroundQueue.push(ProductImportJob, ...)
    

    View Slide

44. throttling
    

    View Slide

45. the right data store
    for the job
    

    View Slide

46. ephemeral data
    sessions carts
    • inventory reservation
    •
    

    View Slide

47. now what?
    

    View Slide

48. catching regressions
    

    View Slide

49. measure it!
    if it moves...
    

    View Slide

50. statsd
    

    View Slide

51. View Slide

52. Liquid::Template.extend StatsD::Instrument
    Liquid::Template.statsd_measure :parse, 'Liquid.Template.parse'
    Liquid::Template.statsd_measure :render, 'Liquid.Template.render'
    

    View Slide

53. PaymentProcessingJob.stats_count :perform,
    'PaymentProcessingJob.processed'
    

    View Slide

54. load testing
    

    View Slide

55. View Slide

56. simulates a flash sale
    

    View Slide

57. several times per week
    

    View Slide

58. slow queries
    

    View Slide

59. # User@Host: shopify[shopify] @ [127.0.0.1]
    # Thread_id: 264419969 Schema: shopify Last_errno: 0 Killed: 0
    # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined:
    147841 Rows_affected: 0 Rows_read: 147841
    # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0
    # InnoDB_trx_id: FF7021AAA
    # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No
    Tmp_table_on_disk: No
    # Filesort: Yes Filesort_on_disk: No Merge_passes: 0
    # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000
    # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000
    # InnoDB_pages_distinct: 475
    SET timestamp=1393385020;
    SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` =
    1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC,
    ends_at DESC LIMIT 1
    

    View Slide

60. determining
    root cause
    

    View Slide

61. https://github.com/snormore/nginx-x-rid-header
    nginx request_id header
    proxy_set_header X-Request-ID "$request_id";
    log_format main '... $request_id'
    step 1
    

    View Slide

62. https://gist.github.com/mnutt/566725
    Complete 200 OK in 100ms (Views: 60ms |
    ActiveRecord: 40ms | request_id=bc12813bce...)
    log_process_action
    ActionController::Instrumentation
    step 2
    

    View Slide

63. https://github.com/basecamp/marginalia
    User Load (0.3ms) SELECT `users`.* FROM `users`
    WHERE `users`.`id` = 1 LIMIT 1
    /*application:Shopify,
    controller:users,action:show,
    request_id:bc12813bce...*/
    basecamp/marginalia
    step 3
    

    View Slide

64. # User@Host: shopify[shopify] @ [127.0.0.1]
    # Thread_id: 264419969 Schema: shopify Last_errno: 0 Killed: 0
    # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined:
    147841 Rows_affected: 0 Rows_read: 147841
    # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0
    # InnoDB_trx_id: FF7021AAA
    # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No
    Tmp_table_on_disk: No
    # Filesort: Yes Filesort_on_disk: No Merge_passes: 0
    # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000
    # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000
    # InnoDB_pages_distinct: 475
    SET timestamp=1393385020;
    SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` =
    1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC,
    ends_at DESC LIMIT 1 /*application:Shopify,controller:orders,action:pay,
    request_id:bc12813bce...*/
    profit!
    

    View Slide

65. access.log
    rails.log
    slow_query.log
    profit! (2)
    

    View Slide

66. schema migration
    with zero downtime
    

    View Slide

67. soundcloud/lhm
    

    View Slide

68. current schema new schema
    

    View Slide

69. insert/delete/update triggers
    

    View Slide

70. INSERT INTO ... SELECT ...
    insert/delete/update triggers
    

    View Slide

71. testing for external calls
    memcached
    •
    mysql •
    redis net/http
    •
    

    View Slide

72. it’s not about preventing
    it’s about raising awareness
    

    View Slide

73. integration test with
    assert_externals(...) do .. end
    Unexpected external call (mysql):
    !"" mysql_load("GiftCard")
    !"" "SELECT `gift_cards`.* FROM `gift_cards` WHERE
    `gift_cards`.`id` = 1063936318 LIMIT 1"
    #"" called from: app/services/gift_card_payment_processing.rb:
    73:in `block in log_successful'
    

    View Slide

74. subscribe('sql.active_record')
    ActiveSupport::Notifications
    ["sql.active_record", 2014-02-26 02:38:43 +0000,
    2014-02-26 02:38:43 +0000, "a119c5ac2aa6fb4a52fe",
    {:sql=>"SELECT `users`.* FROM `users` LIMIT
    1", :name=>"User
    Load", :connection_id=>69893685920420, :binds=>[]}]
    

    View Slide

75. monkey-patch other libs
    to add instrumentation
    

    View Slide

76. thanks! :)
    

    View Slide