Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scaling Shopify

Scaling Shopify

Talk given at ConFoo 2014 on February 28th, 2014.

Christian Joudrey

February 28, 2014
Tweet

More Decks by Christian Joudrey

Other Decks in Technology

Transcript

  1. SCALING SHOPIFY
    ...or ensuring happiness for online shoppers

    View Slide

  2. cjoudrey  
    @

    View Slide

  3. View Slide

  4. View Slide

  5. View Slide

  6. View Slide

  7. the stack

    View Slide

  8. nginx unicorn
    • rails 4

    mysql 5.6 (percona)
    ruby 2.1 •

    View Slide

  9. 95 app servers 3,884 unicorn workers
    5 job servers 387 job workers

    View Slide

  10. 1 request 1 process
    =

    View Slide

  11. scale?

    View Slide

  12. over 90,000 shops

    View Slide

  13. View Slide

  14. 1.6B$ annual GMV
    that’s 3,600$ per min

    View Slide

  15. cyber monday
    black friday

    View Slide

  16. View Slide

  17. 61 M$ in GMV
    in four days

    View Slide

  18. flash sales

    View Slide

  19. View Slide

  20. View Slide

  21. page caching

    View Slide

  22. View Slide

  23. View Slide

  24. shopify/cacheable

    View Slide

  25. generational caching

    View Slide

  26. gzip • etag + 304 not modified

    View Slide

  27. class PostsController < ApplicationController
    def show
    response_cache do
    @post = @shop.posts.find(params[:id])
    respond_with(@post)
    end
    end
    def cache_key_data
    {
    action: action_name,
    format: request.format,
    params: params.slice(:id),
    shop_version: @shop.version
    }
    end
    end

    View Slide

  28. View Slide

  29. flash sale

    View Slide

  30. query caching

    View Slide

  31. shopify/identity_cache

    View Slide

  32. full model caching

    View Slide

  33. opt-in by design

    View Slide

  34. after_commit expiry

    View Slide

  35. class Product < ActiveRecord::Base
    include IdentityCache
    has_many :images
    cache_has_many :images, :embed => true
    end
    product = Product.fetch(id)
    images = product.fetch_images

    View Slide

  36. class Product < ActiveRecord::Base
    include IdentityCache
    cache_index :shop_id, :handle, :unique => true
    end
    Product.fetch_by_shop_id_and_handle(shop_id, handle)

    View Slide

  37. View Slide

  38. flash sale

    View Slide

  39. background jobs

    View Slide

  40. webhooks emails
    • fraud detection

    payment processing

    View Slide

  41. View Slide

  42. priority queues
    payment

    default •
    low realtime

    View Slide

  43. class ProductImportJob
    include BackgroundQueue::Realtime
    def perform(params)
    ...
    end
    end
    BackgroundQueue.push(ProductImportJob, ...)

    View Slide

  44. throttling

    View Slide

  45. the right data store
    for the job

    View Slide

  46. ephemeral data
    sessions carts
    • inventory reservation

    View Slide

  47. now what?

    View Slide

  48. catching regressions

    View Slide

  49. measure it!
    if it moves...

    View Slide

  50. statsd

    View Slide

  51. View Slide

  52. Liquid::Template.extend StatsD::Instrument
    Liquid::Template.statsd_measure :parse, 'Liquid.Template.parse'
    Liquid::Template.statsd_measure :render, 'Liquid.Template.render'

    View Slide

  53. PaymentProcessingJob.stats_count :perform,
    'PaymentProcessingJob.processed'

    View Slide

  54. load testing

    View Slide

  55. View Slide

  56. simulates a flash sale

    View Slide

  57. several times per week

    View Slide

  58. slow queries

    View Slide

  59. # User@Host: shopify[shopify] @ [127.0.0.1]
    # Thread_id: 264419969 Schema: shopify Last_errno: 0 Killed: 0
    # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined:
    147841 Rows_affected: 0 Rows_read: 147841
    # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0
    # InnoDB_trx_id: FF7021AAA
    # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No
    Tmp_table_on_disk: No
    # Filesort: Yes Filesort_on_disk: No Merge_passes: 0
    # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000
    # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000
    # InnoDB_pages_distinct: 475
    SET timestamp=1393385020;
    SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` =
    1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC,
    ends_at DESC LIMIT 1

    View Slide

  60. determining
    root cause

    View Slide

  61. https://github.com/snormore/nginx-x-rid-header
    nginx request_id header
    proxy_set_header X-Request-ID "$request_id";
    log_format main '... $request_id'
    step 1

    View Slide

  62. https://gist.github.com/mnutt/566725
    Complete 200 OK in 100ms (Views: 60ms |
    ActiveRecord: 40ms | request_id=bc12813bce...)
    log_process_action
    ActionController::Instrumentation
    step 2

    View Slide

  63. https://github.com/basecamp/marginalia
    User Load (0.3ms) SELECT `users`.* FROM `users`
    WHERE `users`.`id` = 1 LIMIT 1
    /*application:Shopify,
    controller:users,action:show,
    request_id:bc12813bce...*/
    basecamp/marginalia
    step 3

    View Slide

  64. # User@Host: shopify[shopify] @ [127.0.0.1]
    # Thread_id: 264419969 Schema: shopify Last_errno: 0 Killed: 0
    # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined:
    147841 Rows_affected: 0 Rows_read: 147841
    # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0
    # InnoDB_trx_id: FF7021AAA
    # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No
    Tmp_table_on_disk: No
    # Filesort: Yes Filesort_on_disk: No Merge_passes: 0
    # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000
    # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000
    # InnoDB_pages_distinct: 475
    SET timestamp=1393385020;
    SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` =
    1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC,
    ends_at DESC LIMIT 1 /*application:Shopify,controller:orders,action:pay,
    request_id:bc12813bce...*/
    profit!

    View Slide

  65. access.log
    rails.log
    slow_query.log
    profit! (2)

    View Slide

  66. schema migration
    with zero downtime

    View Slide

  67. soundcloud/lhm

    View Slide

  68. current schema new schema

    View Slide

  69. insert/delete/update triggers

    View Slide

  70. INSERT INTO ... SELECT ...
    insert/delete/update triggers

    View Slide

  71. testing for external calls
    memcached

    mysql •
    redis net/http

    View Slide

  72. it’s not about preventing
    it’s about raising awareness

    View Slide

  73. integration test with
    assert_externals(...) do .. end
    Unexpected external call (mysql):
    !"" mysql_load("GiftCard")
    !"" "SELECT `gift_cards`.* FROM `gift_cards` WHERE
    `gift_cards`.`id` = 1063936318 LIMIT 1"
    #"" called from: app/services/gift_card_payment_processing.rb:
    73:in `block in log_successful'

    View Slide

  74. subscribe('sql.active_record')
    ActiveSupport::Notifications
    ["sql.active_record", 2014-02-26 02:38:43 +0000,
    2014-02-26 02:38:43 +0000, "a119c5ac2aa6fb4a52fe",
    {:sql=>"SELECT `users`.* FROM `users` LIMIT
    1", :name=>"User
    Load", :connection_id=>69893685920420, :binds=>[]}]

    View Slide

  75. monkey-patch other libs
    to add instrumentation

    View Slide

  76. thanks! :)

    View Slide