Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Caching With MessagePack

Caching With MessagePack

RubyKaigi 2022 talk about how we replaced Marshal with MessagePack in the Rails cache of the Shopify monolith.

Chris Salzberg

September 09, 2022
Tweet

More Decks by Chris Salzberg

Other Decks in Programming

Transcript

  1. CACHING
    MSGPACK
    Chris Salzberg
    with
    with

    View Slide

  2. Chris Salzberg
    @shioyama
    Staff Dev @ Ruby & Rails Infra
    Team: Rails in Core

    View Slide

  3. View Slide

  4. View Slide

  5. before After

    View Slide

  6. before After
    cache

    View Slide

  7. let’s talk about
    caching
    caching

    View Slide

  8. Application

    View Slide

  9. Application
    actions

    View Slide

  10. Application
    actions pages

    View Slide

  11. Application
    actions pages fragments

    View Slide

  12. Application
    actions pages fragments queries

    View Slide

  13. Application
    actions pages fragments queries responses

    View Slide

  14. Application
    actions pages fragments queries responses
    file redis memcached
    memory

    View Slide

  15. Application
    actions pages fragments queries responses
    file redis memcached
    memory
    cache

    View Slide

  16. Application
    actions pages fragments queries responses
    file redis memcached
    memory
    cache cache

    View Slide

  17. Application
    actions pages fragments queries responses
    file redis memcached
    memory
    cache cache cache ...

    View Slide

  18. View Slide

  19. Application
    actions pages fragments queries responses

    View Slide

  20. Application
    actions pages fragments queries responses
    Cache

    View Slide

  21. Application
    actions pages fragments queries
    file redis memcached
    memory
    Cache
    responses

    View Slide

  22. file redis memcached
    memory
    Application
    actions pages fragments queries
    Cache
    responses
    Marshal

    View Slide

  23. View Slide

  24. irb(main):010:0> Marshal.method(:dump).source

    View Slide

  25. irb(main):010:0> Marshal.method(:dump).source
    Could not locate source for dump!
    (MethodSource::SourceNotFoundError)

    View Slide

  26. let’s talk about
    Marshal
    Marshal

    View Slide

  27. class Post < ApplicationRecord
    end

    View Slide

  28. post = Post.create(title: "Caching With MessagePack")
    #=> #

    View Slide

  29. post = Post.create(title: "Caching With MessagePack")
    #=> #serialized_post = Marshal.dump(post)

    View Slide

  30. 1607 bytes!!!
    post = Post.create(title: "Caching With MessagePack")
    #=> #serialized_post = Marshal.dump(post)
    #=> "\x04\bo:\tPost\x1A:\x10@new_recordF:\x10@attributeso:\x1EActiveModel::AttributeSet\x06;\a{\tI\"\aid\x06:\x06ETo:)ActiveMod
    el::Attribute::FromDatabase\n:\n@name@\b:\x1C@value_before_type_casti\x06:\n@typeo:EActiveRecord::ConnectionAdapters::SQLite3Ad
    apter::SQLite3Integer\t:\x0F@precision0:\v@scale0:\v@limit0:\v@rangeo:\nRange\b:\texclT:\nbeginl-\t\x00\x00\x00\x00\x00\x00\x00
    \x80:\bendl+\t\x00\x00\x00\x00\x00\x00\x00\x80:\x18@original_attribute0:\v@valuei\x06I\"\ntitle\x06;\tTo;\n\n;\v@\x0E;\fI\"\x1C
    Caching With MessagePack\x06;\tT;\ro:\x1EActiveModel::Type::String\n:\n@trueI\"\x06t\x06;\tT:\v@falseI\"\x06f\x06;\tT;\x0F0;\x1
    00;\x110;\x170;\x18I\"\x1CCaching With MessagePack\x06;\tTI\"\x0Fcreated_at\x06;\tTo;\n\n;\v@\x15;\fU: ActiveSupport::TimeWithZ
    one[\bIu:\tTime\re\x8F\x1E\xC0\xA7\x88\x83\x02\x06:\tzoneI\"\bUTC\x06;\tFI\"\bUTC\x06;\tTIu;\x1D\re\x8F\x1E\xC0\xA7\x88\x83\x02
    \x06;\x1E@\x19;\rU:JActiveRecord::AttributeMethods::TimeZoneConversion::TimeZoneConverter[\t:\v__v2__[\x00[\x00o:!ActiveRecord:
    :Type::DateTime\b;\x0Fi\v;\x100;\x110;\x170;\x18@\x17I\"\x0Fupdated_at\x06;\tTo;\n\n;\v@\";\fU;\x1C[\b@\x1A@\eIu;\x1D\re\x8F\x1
    E\xC0\xA7\x88\x83\x02\x06;\x1E@\x19;\rU;\x1F[\t; [\x00[\x00@!;\x170;\x18@$:\x17@association_cache{\x00:\x0E@readonlyF:\e@previo
    usly_new_recordT:\x0F@destroyedF:\x1C@marked_for_destructionF:\x1E@destroyed_by_association0:\x1E@_start_transaction_state0:\x1
    1@primary_key@\b:\x14@strict_loadingF:\x19@strict_loading_mode:\ball:$@_new_record_before_last_commitT:\x18@validation_context0
    :\f@errorso:\x18ActiveModel::Errors\a:\n@base@\x00;/[\x00:\x13@_touch_recordT:\x1D@mutations_from_database0: @mutations_before_
    last_saveo:*ActiveModel::AttributeMutationTracker\x06;\ao;\b\x06;\a{\t@\bo:%ActiveModel::Attribute::FromUser\n;\v@\b;\fi\x06;\r
    @\n;\x17o;\n\n;\v@\b;\f0;\r@\n;\x170;\x180;\x18i\x06@\x0Eo;6\n;\v@\x0E;\fI\"\x1CCaching With MessagePack\x06;\tT;\r@\x11;\x17o;
    \n\t;\v@\x0E;\f0;\r@\x11;\x170;\x18@\x10@\x15o;6\n;\v@\x15;\f@\x1A;\r@\x1D;\x17o;\n\n;\v@\x15;\f0;\r@\x1D;\x170;\x180;\x18@\x17
    @\"o;6\n;\v@\";\f@\x1A;\r@';\x17o;\n\n;\v@\";\f0;\r@';\x170;\x180;\x18@$:\x1F@_committed_already_calledF:\x1F@_trigger_destroy_
    callbackF:\x1E@_trigger_update_callbackF"

    View Slide

  31. post = Post.create(title: "Caching With MessagePack")
    #=> #serialized_post = Marshal.dump(post)
    #=> "\x04\bo:\tPost\x1A:\x10@new_recordF:\x10@attributeso:\x1EActiveModel::AttributeSet\x06;\a{\tI\"\aid\x06:\x06ETo:)ActiveMod
    el::Attribute::FromDatabase\n:\n@name@\b:\x1C@value_before_type_casti\x06:\n@typeo:EActiveRecord::ConnectionAdapters::SQLite3Ad
    apter::SQLite3Integer\t:\x0F@precision0:\v@scale0:\v@limit0:\v@rangeo:\nRange\b:\texclT:\nbeginl-\t\x00\x00\x00\x00\x00\x00\x00
    \x80:\bendl+\t\x00\x00\x00\x00\x00\x00\x00\x80:\x18@original_attribute0:\v@valuei\x06I\"\ntitle\x06;\tTo;\n\n;\v@\x0E;\fI\"\x1C
    Caching With MessagePack\x06;\tT;\ro:\x1EActiveModel::Type::String\n:\n@trueI\"\x06t\x06;\tT:\v@falseI\"\x06f\x06;\tT;\x0F0;\x1
    00;\x110;\x170;\x18I\"\x1CCaching With MessagePack\x06;\tTI\"\x0Fcreated_at\x06;\tTo;\n\n;\v@\x15;\fU: ActiveSupport::TimeWithZ
    one[\bIu:\tTime\re\x8F\x1E\xC0\xA7\x88\x83\x02\x06:\tzoneI\"\bUTC\x06;\tFI\"\bUTC\x06;\tTIu;\x1D\re\x8F\x1E\xC0\xA7\x88\x83\x02
    \x06;\x1E@\x19;\rU:JActiveRecord::AttributeMethods::TimeZoneConversion::TimeZoneConverter[\t:\v__v2__[\x00[\x00o:!ActiveRecord:
    :Type::DateTime\b;\x0Fi\v;\x100;\x110;\x170;\x18@\x17I\"\x0Fupdated_at\x06;\tTo;\n\n;\v@\";\fU;\x1C[\b@\x1A@\eIu;\x1D\re\x8F\x1
    E\xC0\xA7\x88\x83\x02\x06;\x1E@\x19;\rU;\x1F[\t; [\x00[\x00@!;\x170;\x18@$:\x17@association_cache{\x00:\x0E@readonlyF:\e@previo
    usly_new_recordT:\x0F@destroyedF:\x1C@marked_for_destructionF:\x1E@destroyed_by_association0:\x1E@_start_transaction_state0:\x1
    1@primary_key@\b:\x14@strict_loadingF:\x19@strict_loading_mode:\ball:$@_new_record_before_last_commitT:\x18@validation_context0
    :\f@errorso:\x18ActiveModel::Errors\a:\n@base@\x00;/[\x00:\x13@_touch_recordT:\x1D@mutations_from_database0: @mutations_before_
    last_saveo:*ActiveModel::AttributeMutationTracker\x06;\ao;\b\x06;\a{\t@\bo:%ActiveModel::Attribute::FromUser\n;\v@\b;\fi\x06;\r
    @\n;\x17o;\n\n;\v@\b;\f0;\r@\n;\x170;\x180;\x18i\x06@\x0Eo;6\n;\v@\x0E;\fI\"\x1CCaching With MessagePack\x06;\tT;\r@\x11;\x17o;
    \n\t;\v@\x0E;\f0;\r@\x11;\x170;\x18@\x10@\x15o;6\n;\v@\x15;\f@\x1A;\r@\x1D;\x17o;\n\n;\v@\x15;\f0;\r@\x1D;\x170;\x180;\x18@\x17
    @\"o;6\n;\v@\";\f@\x1A;\r@';\x17o;\n\n;\v@\";\f0;\r@';\x170;\x180;\x18@$:\x1F@_committed_already_calledF:\x1F@_trigger_destroy_
    callbackF:\x1E@_trigger_update_callbackF"
    Constants

    View Slide

  32. post = Post.create(title: "Caching With MessagePack")
    #=> #serialized_post = Marshal.dump(post)
    #=> "\x04\bo:\tPost\x1A:\x10@new_recordF:\x10@attributeso:\x1EActiveModel::AttributeSet\x06;\a{\tI\"\aid\x06:\x06ETo:)ActiveMod
    el::Attribute::FromDatabase\n:\n@name@\b:\x1C@value_before_type_casti\x06:\n@typeo:EActiveRecord::ConnectionAdapters::SQLite3Ad
    apter::SQLite3Integer\t:\x0F@precision0:\v@scale0:\v@limit0:\v@rangeo:\nRange\b:\texclT:\nbeginl-\t\x00\x00\x00\x00\x00\x00\x00
    \x80:\bendl+\t\x00\x00\x00\x00\x00\x00\x00\x80:\x18@original_attribute0:\v@valuei\x06I\"\ntitle\x06;\tTo;\n\n;\v@\x0E;\fI\"\x1C
    Caching With MessagePack\x06;\tT;\ro:\x1EActiveModel::Type::String\n:\n@trueI\"\x06t\x06;\tT:\v@falseI\"\x06f\x06;\tT;\x0F0;\x1
    00;\x110;\x170;\x18I\"\x1CCaching With MessagePack\x06;\tTI\"\x0Fcreated_at\x06;\tTo;\n\n;\v@\x15;\fU: ActiveSupport::TimeWithZ
    one[\bIu:\tTime\re\x8F\x1E\xC0\xA7\x88\x83\x02\x06:\tzoneI\"\bUTC\x06;\tFI\"\bUTC\x06;\tTIu;\x1D\re\x8F\x1E\xC0\xA7\x88\x83\x02
    \x06;\x1E@\x19;\rU:JActiveRecord::AttributeMethods::TimeZoneConversion::TimeZoneConverter[\t:\v__v2__[\x00[\x00o:!ActiveRecord:
    :Type::DateTime\b;\x0Fi\v;\x100;\x110;\x170;\x18@\x17I\"\x0Fupdated_at\x06;\tTo;\n\n;\v@\";\fU;\x1C[\b@\x1A@\eIu;\x1D\re\x8F\x1
    E\xC0\xA7\x88\x83\x02\x06;\x1E@\x19;\rU;\x1F[\t; [\x00[\x00@!;\x170;\x18@$:\x17@association_cache{\x00:\x0E@readonlyF:\e@previo
    usly_new_recordT:\x0F@destroyedF:\x1C@marked_for_destructionF:\x1E@destroyed_by_association0:\x1E@_start_transaction_state0:\x1
    1@primary_key@\b:\x14@strict_loadingF:\x19@strict_loading_mode:\ball:$@_new_record_before_last_commitT:\x18@validation_context0
    :\f@errorso:\x18ActiveModel::Errors\a:\n@base@\x00;/[\x00:\x13@_touch_recordT:\x1D@mutations_from_database0: @mutations_before_
    last_saveo:*ActiveModel::AttributeMutationTracker\x06;\ao;\b\x06;\a{\t@\bo:%ActiveModel::Attribute::FromUser\n;\v@\b;\fi\x06;\r
    @\n;\x17o;\n\n;\v@\b;\f0;\r@\n;\x170;\x180;\x18i\x06@\x0Eo;6\n;\v@\x0E;\fI\"\x1CCaching With MessagePack\x06;\tT;\r@\x11;\x17o;
    \n\t;\v@\x0E;\f0;\r@\x11;\x170;\x18@\x10@\x15o;6\n;\v@\x15;\f@\x1A;\r@\x1D;\x17o;\n\n;\v@\x15;\f0;\r@\x1D;\x170;\x180;\x18@\x17
    @\"o;6\n;\v@\";\f@\x1A;\r@';\x17o;\n\n;\v@\";\f0;\r@';\x170;\x180;\x18@$:\x1F@_committed_already_calledF:\x1F@_trigger_destroy_
    callbackF:\x1E@_trigger_update_callbackF"
    ivars

    View Slide

  33. post = Post.create(title: "Caching With MessagePack")
    #=> #serialized_post = Marshal.dump(post)
    #=> "\x04\bo:\tPost\x1A:\x10@new_recordF:\x10@attributeso:\x1EActiveModel::AttributeSet\x06;\a{\tI\"\aid\x06:\x06ETo:)ActiveMod
    el::Attribute::FromDatabase\n:\n@name@\b:\x1C@value_before_type_casti\x06:\n@typeo:EActiveRecord::ConnectionAdapters::SQLite3Ad
    apter::SQLite3Integer\t:\x0F@precision0:\v@scale0:\v@limit0:\v@rangeo:\nRange\b:\texclT:\nbeginl-\t\x00\x00\x00\x00\x00\x00\x00
    \x80:\bendl+\t\x00\x00\x00\x00\x00\x00\x00\x80:\x18@original_attribute0:\v@valuei\x06I\"\ntitle\x06;\tTo;\n\n;\v@\x0E;\fI\"\x1C
    Caching With MessagePack\x06;\tT;\ro:\x1EActiveModel::Type::String\n:\n@trueI\"\x06t\x06;\tT:\v@falseI\"\x06f\x06;\tT;\x0F0;\x1
    00;\x110;\x170;\x18I\"\x1CCaching With MessagePack\x06;\tTI\"\x0Fcreated_at\x06;\tTo;\n\n;\v@\x15;\fU: ActiveSupport::TimeWithZ
    one[\bIu:\tTime\re\x8F\x1E\xC0\xA7\x88\x83\x02\x06:\tzoneI\"\bUTC\x06;\tFI\"\bUTC\x06;\tTIu;\x1D\re\x8F\x1E\xC0\xA7\x88\x83\x02
    \x06;\x1E@\x19;\rU:JActiveRecord::AttributeMethods::TimeZoneConversion::TimeZoneConverter[\t:\v__v2__[\x00[\x00o:!ActiveRecord:
    :Type::DateTime\b;\x0Fi\v;\x100;\x110;\x170;\x18@\x17I\"\x0Fupdated_at\x06;\tTo;\n\n;\v@\";\fU;\x1C[\b@\x1A@\eIu;\x1D\re\x8F\x1
    E\xC0\xA7\x88\x83\x02\x06;\x1E@\x19;\rU;\x1F[\t; [\x00[\x00@!;\x170;\x18@$:\x17@association_cache{\x00:\x0E@readonlyF:\e@previo
    usly_new_recordT:\x0F@destroyedF:\x1C@marked_for_destructionF:\x1E@destroyed_by_association0:\x1E@_start_transaction_state0:\x1
    1@primary_key@\b:\x14@strict_loadingF:\x19@strict_loading_mode:\ball:$@_new_record_before_last_commitT:\x18@validation_context0
    :\f@errorso:\x18ActiveModel::Errors\a:\n@base@\x00;/[\x00:\x13@_touch_recordT:\x1D@mutations_from_database0: @mutations_before_
    last_saveo:*ActiveModel::AttributeMutationTracker\x06;\ao;\b\x06;\a{\t@\bo:%ActiveModel::Attribute::FromUser\n;\v@\b;\fi\x06;\r
    @\n;\x17o;\n\n;\v@\b;\f0;\r@\n;\x170;\x180;\x18i\x06@\x0Eo;6\n;\v@\x0E;\fI\"\x1CCaching With MessagePack\x06;\tT;\r@\x11;\x17o;
    \n\t;\v@\x0E;\f0;\r@\x11;\x170;\x18@\x10@\x15o;6\n;\v@\x15;\f@\x1A;\r@\x1D;\x17o;\n\n;\v@\x15;\f0;\r@\x1D;\x170;\x180;\x18@\x17
    @\"o;6\n;\v@\";\f@\x1A;\r@';\x17o;\n\n;\v@\";\f0;\r@';\x170;\x180;\x18@$:\x1F@_committed_already_calledF:\x1F@_trigger_destroy_
    callbackF:\x1E@_trigger_update_callbackF"
    Values

    View Slide

  34. post = Post.create(title: "Caching With MessagePack")
    #=> #serialized_post = Marshal.dump(post)
    Marshal.load(serialized_post)
    #=> ## id: 1,
    # title: "Caching With MessagePack",
    # created_at: Wed, 27 Apr 2022 05:00:40.231591000 UTC +00:00,
    # updated_at: Wed, 27 Apr 2022 05:00:40.231591000 UTC +00:00>

    View Slide

  35. marshal encodes
    the universe
    the universe

    View Slide

  36. marshal.c
    #define MARSHAL_MAJOR 4
    #define MARSHAL_MINOR 8
    #define TYPE_NIL '0'
    #define TYPE_TRUE 'T'
    #define TYPE_FALSE 'F'
    #define TYPE_FIXNUM 'i'
    #define TYPE_BIGNUM 'l'
    #define TYPE_FLOAT 'f'
    #define TYPE_SYMBOL ':'
    #define TYPE_CLASS 'c'
    #define TYPE_MODULE 'm'
    #define TYPE_LINK '@'
    #define TYPE_SYMLINK ';'
    #define TYPE_STRING '"'
    #define TYPE_REGEXP '/'
    #define TYPE_ARRAY '['
    #define TYPE_HASH '{'
    #define TYPE_HASH_DEF '}'
    #define TYPE_STRUCT 'S'
    #define TYPE_OBJECT 'o'
    #define TYPE_IVAR 'I'
    #define TYPE_UCLASS 'C'
    #define TYPE_EXTENDED 'e'
    #define TYPE_USERDEF 'u'
    #define TYPE_USRMARSHAL 'U'

    View Slide

  37. marshal.c
    #define MARSHAL_MAJOR 4
    #define MARSHAL_MINOR 8
    #define TYPE_NIL '0'
    #define TYPE_TRUE 'T'
    #define TYPE_FALSE 'F'
    #define TYPE_FIXNUM 'i'
    #define TYPE_BIGNUM 'l'
    #define TYPE_FLOAT 'f'
    #define TYPE_SYMBOL ':'
    #define TYPE_CLASS 'c'
    #define TYPE_MODULE 'm'
    #define TYPE_LINK '@'
    #define TYPE_SYMLINK ';'
    #define TYPE_STRING '"'
    #define TYPE_REGEXP '/'
    #define TYPE_ARRAY '['
    #define TYPE_HASH '{'
    #define TYPE_HASH_DEF '}'
    #define TYPE_STRUCT 'S'
    #define TYPE_OBJECT 'o'
    #define TYPE_IVAR 'I'
    #define TYPE_UCLASS 'C'
    #define TYPE_EXTENDED 'e'
    #define TYPE_USERDEF 'u'
    #define TYPE_USRMARSHAL 'U'

    View Slide

  38. marshal.c
    #define MARSHAL_MAJOR 4
    #define MARSHAL_MINOR 8
    #define TYPE_NIL '0'
    #define TYPE_TRUE 'T'
    #define TYPE_FALSE 'F'
    #define TYPE_FIXNUM 'i'
    #define TYPE_BIGNUM 'l'
    #define TYPE_FLOAT 'f'
    #define TYPE_SYMBOL ':'
    #define TYPE_CLASS 'c'
    #define TYPE_MODULE 'm'
    #define TYPE_LINK '@'
    #define TYPE_SYMLINK ';'
    #define TYPE_STRING '"'
    #define TYPE_REGEXP '/'
    #define TYPE_ARRAY '['
    #define TYPE_HASH '{'
    #define TYPE_HASH_DEF '}'
    #define TYPE_STRUCT 'S'
    #define TYPE_OBJECT 'o'
    #define TYPE_IVAR 'I'
    #define TYPE_UCLASS 'C'
    #define TYPE_EXTENDED 'e'
    #define TYPE_USERDEF 'u'
    #define TYPE_USRMARSHAL 'U'
    atomic

    View Slide

  39. marshal.c
    #define MARSHAL_MAJOR 4
    #define MARSHAL_MINOR 8
    #define TYPE_NIL '0'
    #define TYPE_TRUE 'T'
    #define TYPE_FALSE 'F'
    #define TYPE_FIXNUM 'i'
    #define TYPE_BIGNUM 'l'
    #define TYPE_FLOAT 'f'
    #define TYPE_SYMBOL ':'
    #define TYPE_CLASS 'c'
    #define TYPE_MODULE 'm'
    #define TYPE_LINK '@'
    #define TYPE_SYMLINK ';'
    #define TYPE_STRING '"'
    #define TYPE_REGEXP '/'
    #define TYPE_ARRAY '['
    #define TYPE_HASH '{'
    #define TYPE_HASH_DEF '}'
    #define TYPE_STRUCT 'S'
    #define TYPE_OBJECT 'o'
    #define TYPE_IVAR 'I'
    #define TYPE_UCLASS 'C'
    #define TYPE_EXTENDED 'e'
    #define TYPE_USERDEF 'u'
    #define TYPE_USRMARSHAL 'U'
    atomic
    composite

    View Slide

  40. marshal.c
    #define MARSHAL_MAJOR 4
    #define MARSHAL_MINOR 8
    #define TYPE_NIL '0'
    #define TYPE_TRUE 'T'
    #define TYPE_FALSE 'F'
    #define TYPE_FIXNUM 'i'
    #define TYPE_BIGNUM 'l'
    #define TYPE_FLOAT 'f'
    #define TYPE_SYMBOL ':'
    #define TYPE_CLASS 'c'
    #define TYPE_MODULE 'm'
    #define TYPE_LINK '@'
    #define TYPE_SYMLINK ';'
    #define TYPE_STRING '"'
    #define TYPE_REGEXP '/'
    #define TYPE_ARRAY '['
    #define TYPE_HASH '{'
    #define TYPE_HASH_DEF '}'
    #define TYPE_STRUCT 'S'
    #define TYPE_OBJECT 'o'
    #define TYPE_IVAR 'I'
    #define TYPE_UCLASS 'C'
    #define TYPE_EXTENDED 'e'
    #define TYPE_USERDEF 'u'
    #define TYPE_USRMARSHAL 'U'
    atomic
    composite

    View Slide

  41. Objects
    Objects

    View Slide

  42. marshal.c
    #define MARSHAL_MAJOR 4
    #define MARSHAL_MINOR 8
    #define TYPE_NIL '0'
    #define TYPE_TRUE 'T'
    #define TYPE_FALSE 'F'
    #define TYPE_FIXNUM 'i'
    #define TYPE_BIGNUM 'l'
    #define TYPE_FLOAT 'f'
    #define TYPE_SYMBOL ':'
    #define TYPE_CLASS 'c'
    #define TYPE_MODULE 'm'
    #define TYPE_LINK '@'
    #define TYPE_SYMLINK ';'
    #define TYPE_STRING '"'
    #define TYPE_REGEXP '/'
    #define TYPE_ARRAY '['
    #define TYPE_HASH '{'
    #define TYPE_HASH_DEF '}'
    #define TYPE_STRUCT 'S'
    #define TYPE_OBJECT 'o'
    #define TYPE_IVAR 'I'
    #define TYPE_UCLASS 'C'
    #define TYPE_EXTENDED 'e'
    #define TYPE_USERDEF 'u'
    #define TYPE_USRMARSHAL 'U'
    atomic
    composite

    View Slide

  43. \x04\bo:\tPost\x1A:\x10@new_recordF:\x10@attributeso:\x1EActiveModel::Attribu
    teSet\x06;\a{\tI\"\aid\x06:\x06ETo:)ActiveModel::Attribute::FromDatabase\n:\
    n@name@\b:\x1C@value_before_type_casti\x06:\n@typeo:EActiveRecord::Connection
    Adapters::SQLite3Adapter::SQLite3Integer\t:\x0F@precision0:\v@scale0:\v@limit
    0:\v@rangeo:\nRange\b:\texclT:\nbeginl-\t\x00\x00\x00\x00\x00\x00\x00\x80:\be
    ndl+\t\x00\x00\x00\x00\x00\x00\x00\x80:\x18@original_attribute0:\v@valuei\x06
    I\"\ntitle\x06;\tTo;\n\n;\v@\x0E;\fI\"\x1CCaching With MessagePack\x06;\tT;\
    ro :\x1EActiveModel::Type::String\n:\n@trueI\"\x06t\x06;\tT:\v@falseI\"\x06f\
    x06 ;\tT;\x0F0;\x100;\x110;\x170;\x18I\"\x1CCaching With MessagePack\x06;\
    tTI\"\x0 Fcreated_at\x06;\tTo;\n\n;\v@\x15;\fU: ...

    View Slide

  44. 0408 6f3a 0950 6f73 741a 3a10 406e 6577 5f72 6563
    6f72 6446 3a10 4061 7474 7269 6275 7465 736f 3a1e
    4163 7469 7665 4d6f 6465 6c3a 3a41 7474 7269 6275
    7465 5365 7406 3b07 7b09 4922 0769 6406 3a06 4554
    6f3a 2941 6374 6976 654d 6f64 656c 3a3a...
    \x04\bo:\tPost\x1A:\x10@new_recordF:\x10@attributeso:\x1EActiveModel::Attribu
    teSet\x06;\a{\tI\"\aid\x06:\x06ETo:)ActiveModel::Attribute::FromDatabase\n:\
    n@name@\b:\x1C@value_before_type_casti\x06:\n@typeo:EActiveRecord::Connection
    Adapters::SQLite3Adapter::SQLite3Integer\t:\x0F@precision0:\v@scale0:\v@limit
    0:\v@rangeo:\nRange\b:\texclT:\nbeginl-\t\x00\x00\x00\x00\x00\x00\x00\x80:\be
    ndl+\t\x00\x00\x00\x00\x00\x00\x00\x80:\x18@original_attribute0:\v@valuei\x06
    I\"\ntitle\x06;\tTo;\n\n;\v@\x0E;\fI\"\x1CCaching With MessagePack\x06;\tT;\
    ro :\x1EActiveModel::Type::String\n:\n@trueI\"\x06t\x06;\tT:\v@falseI\"\x06f\
    x06 ;\tT;\x0F0;\x100;\x110;\x170;\x18I\"\x1CCaching With MessagePack\x06;\
    tTI\"\x0 Fcreated_at\x06;\tTo;\n\n;\v@\x15;\fU: ...

    View Slide

  45. 0408 6f3a 0950 6f73 741a 3a10 406e 6577 5f72 6563
    6f72 6446 3a10 4061 7474 7269 6275 7465 736f 3a1e
    4163 7469 7665 4d6f 6465 6c3a 3a41 7474 7269 6275
    7465 5365 7406 3b07 7b09 4922 0769 6406 3a06 4554
    6f3a 2941 6374 6976 654d 6f64 656c 3a3a...
    Version

    View Slide

  46. 0408 6f3a 0950 6f73 741a 3a10 406e 6577 5f72 6563
    6f72 6446 3a10 4061 7474 7269 6275 7465 736f 3a1e
    4163 7469 7665 4d6f 6465 6c3a 3a41 7474 7269 6275
    7465 5365 7406 3b07 7b09 4922 0769 6406 3a06 4554
    6f3a 2941 6374 6976 654d 6f64 656c 3a3a...
    o :
    length
    o s t
    P
    number of ivars

    View Slide

  47. 0408 6f3a 0950 6f73 741a 3a10 406e 6577 5f72 6563
    6f72 6446 3a10 4061 7474 7269 6275 7465 736f 3a1e
    4163 7469 7665 4d6f 6465 6c3a 3a41 7474 7269 6275
    7465 5365 7406 3b07 7b09 4922 0769 6406 3a06 4554
    6f3a 2941 6374 6976 654d 6f64 656c 3a3a...
    :
    length
    @new_record

    View Slide

  48. 0408 6f3a 0950 6f73 741a 3a10 406e 6577 5f72 6563
    6f72 6446 3a10 4061 7474 7269 6275 7465 736f 3a1e
    4163 7469 7665 4d6f 6465 6c3a 3a41 7474 7269 6275
    7465 5365 7406 3b07 7b09 4922 0769 6406 3a06 4554
    6f3a 2941 6374 6976 654d 6f64 656c 3a3a...
    F

    View Slide

  49. 0408 6f3a 0950 6f73 741a 3a10 406e 6577 5f72 6563
    6f72 6446 3a10 4061 7474 7269 6275 7465 736f 3a1e
    4163 7469 7665 4d6f 6465 6c3a 3a41 7474 7269 6275
    7465 5365 7406 3b07 7b09 4922 0769 6406 3a06 4554
    6f3a 2941 6374 6976 654d 6f64 656c 3a3a...

    View Slide

  50. variables
    variables

    View Slide

  51. mystring = "Caching With MessagePack"

    View Slide

  52. mystring = "Caching With MessagePack"
    mystring.instance_variable_set(:@hidden, true)
    mystring.instance_variable_get(:@hidden)
    #=> true

    View Slide

  53. mystring = "Caching With MessagePack"
    mystring.instance_variable_set(:@hidden, true)
    mystring.instance_variable_get(:@hidden)
    #=> true
    encoded = Marshal.dump(mystring)
    decoded = Marshal.load(encoded)
    decoded.instance_variable_get(:@hidden)
    #=> ???

    View Slide

  54. mystring = "Caching With MessagePack"
    mystring.instance_variable_set(:@hidden, true)
    mystring.instance_variable_get(:@hidden)
    #=> true
    encoded = Marshal.dump(mystring)
    decoded = Marshal.load(encoded)
    decoded.instance_variable_get(:@hidden)
    #=> true

    View Slide

  55. marshal.c
    #define MARSHAL_MAJOR 4
    #define MARSHAL_MINOR 8
    #define TYPE_NIL '0'
    #define TYPE_TRUE 'T'
    #define TYPE_FALSE 'F'
    #define TYPE_FIXNUM 'i'
    #define TYPE_BIGNUM 'l'
    #define TYPE_FLOAT 'f'
    #define TYPE_SYMBOL ':'
    #define TYPE_CLASS 'c'
    #define TYPE_MODULE 'm'
    #define TYPE_LINK '@'
    #define TYPE_SYMLINK ';'
    #define TYPE_STRING '"'
    #define TYPE_REGEXP '/'
    #define TYPE_ARRAY '['
    #define TYPE_HASH '{'
    #define TYPE_HASH_DEF '}'
    #define TYPE_STRUCT 'S'
    #define TYPE_OBJECT 'o'
    #define TYPE_IVAR 'I'
    #define TYPE_UCLASS 'C'
    #define TYPE_EXTENDED 'e'
    #define TYPE_USERDEF 'u'
    #define TYPE_USRMARSHAL 'U'
    atomic
    composite

    View Slide

  56. links
    links

    View Slide

  57. myarray = []

    View Slide

  58. myarray = []
    myarray << myarray
    #=> [[...]]

    View Slide

  59. myarray = []
    myarray << myarray
    #=> [[...]]
    decoded = Marshal.load(Marshal.dump(myarray))
    #=> ???

    View Slide

  60. myarray = []
    myarray << myarray
    #=> [[...]]
    decoded = Marshal.load(Marshal.dump(myarray))
    #=> [[...]]

    View Slide

  61. myarray = []
    myarray << myarray
    #=> [[...]]
    decoded = Marshal.load(Marshal.dump(myarray))
    #=> [[...]]
    decoded.first == decoded
    #=> true

    View Slide

  62. marshal.c
    #define MARSHAL_MAJOR 4
    #define MARSHAL_MINOR 8
    #define TYPE_NIL '0'
    #define TYPE_TRUE 'T'
    #define TYPE_FALSE 'F'
    #define TYPE_FIXNUM 'i'
    #define TYPE_BIGNUM 'l'
    #define TYPE_FLOAT 'f'
    #define TYPE_SYMBOL ':'
    #define TYPE_CLASS 'c'
    #define TYPE_MODULE 'm'
    #define TYPE_LINK '@'
    #define TYPE_SYMLINK ';'
    #define TYPE_STRING '"'
    #define TYPE_REGEXP '/'
    #define TYPE_ARRAY '['
    #define TYPE_HASH '{'
    #define TYPE_HASH_DEF '}'
    #define TYPE_STRUCT 'S'
    #define TYPE_OBJECT 'o'
    #define TYPE_IVAR 'I'
    #define TYPE_UCLASS 'C'
    #define TYPE_EXTENDED 'e'
    #define TYPE_USERDEF 'u'
    #define TYPE_USRMARSHAL 'U'
    atomic
    composite

    View Slide

  63. 0408 5b06 4000

    View Slide

  64. 0408 5b06 4000
    [
    length
    TYPE_ARRAY

    View Slide

  65. 0408 5b06 4000
    [
    length
    @
    position
    TYPE_LINK

    View Slide

  66. core type
    core type
    subclasses
    subclasses

    View Slide

  67. class MyHash < Hash
    end

    View Slide

  68. class MyHash < Hash
    end
    my_hash = MyHash.new
    #=> {}

    View Slide

  69. class MyHash < Hash
    end
    my_hash = MyHash.new
    my_hash.class
    #=> MyHash

    View Slide

  70. class MyHash < Hash
    end
    my_hash = MyHash.new
    encoded = Marshal.dump(my_hash)
    decoded = Marshal.load(encoded)
    #=> {}

    View Slide

  71. class MyHash < Hash
    end
    my_hash = MyHash.new
    encoded = Marshal.dump(my_hash)
    decoded = Marshal.load(encoded)
    decoded.class
    #=> MyHash

    View Slide

  72. marshal.c
    #define MARSHAL_MAJOR 4
    #define MARSHAL_MINOR 8
    #define TYPE_NIL '0'
    #define TYPE_TRUE 'T'
    #define TYPE_FALSE 'F'
    #define TYPE_FIXNUM 'i'
    #define TYPE_BIGNUM 'l'
    #define TYPE_FLOAT 'f'
    #define TYPE_SYMBOL ':'
    #define TYPE_CLASS 'c'
    #define TYPE_MODULE 'm'
    #define TYPE_LINK '@'
    #define TYPE_SYMLINK ';'
    #define TYPE_STRING '"'
    #define TYPE_REGEXP '/'
    #define TYPE_ARRAY '['
    #define TYPE_HASH '{'
    #define TYPE_HASH_DEF '}'
    #define TYPE_STRUCT 'S'
    #define TYPE_OBJECT 'o'
    #define TYPE_IVAR 'I'
    #define TYPE_UCLASS 'C'
    #define TYPE_EXTENDED 'e'
    #define TYPE_USERDEF 'u'
    #define TYPE_USRMARSHAL 'U'
    atomic
    composite

    View Slide

  73. MARSHAL
    MARSHAL
    RUBY DEVELOPER
    RUBY DEVELOPER

    View Slide

  74. let’s talk about
    msgpack
    msgpack

    View Slide

  75. before After
    cache

    View Slide

  76. before After

    View Slide

  77. We need a format that
    does not encode the universe.

    View Slide

  78. View Slide

  79. encoded = MessagePack.pack({"foo" => "bar"})

    View Slide

  80. encoded = MessagePack.pack({"foo" => "bar"})
    => "\x81\xA3foo\xA3bar"

    View Slide

  81. encoded = MessagePack.pack({"foo" => "bar"})
    => "\x81\xA3foo\xA3bar"
    decoded = MessagePack.unpack(encoded)
    => {"foo"=>"bar"}

    View Slide

  82. Core Types

    View Slide

  83. Core Types
    atomic

    View Slide

  84. Core Types
    composite

    View Slide

  85. Core Types
    No Object Type

    View Slide

  86. Core Types
    No Object Type
    No ivar Type

    View Slide

  87. 0408 4922 0866 6f6f 063a 0645 54
    Marshal.dump("foo")

    View Slide

  88. 0408 4922 0866 6f6f 063a 0645 54
    Marshal.dump("foo")
    foo
    "
    length 3

    View Slide

  89. 0408 4922 0866 6f6f 063a 0645 54
    Marshal.dump("foo")
    :E
    TYPE_IVAR
    length 1
    T
    “This string’s encoding is UTF-8.”

    View Slide

  90. a366 6f6f
    MessagePack.pack("foo")

    View Slide

  91. a366 6f6f
    MessagePack.pack("foo")
    foo

    View Slide

  92. a366 6f6f
    MessagePack.pack("foo")
    foo
    (String type encoding is UTF-8.)

    View Slide

  93. a366 6f6f
    MessagePack.pack("foo")
    10100011
    string of length < 31 bytes

    View Slide

  94. a366 6f6f
    MessagePack.pack("foo")
    10100011
    length 3

    View Slide

  95. View Slide

  96. ActiveRecord::Base

    View Slide

  97. Object?

    View Slide

  98. We need a format that
    does not encode the universe.
    whole
    ^

    View Slide

  99. We need a format that
    does not encode the universe.
    (but let’s us encode parts of it)
    whole
    ^

    View Slide

  100. factory = MessagePack::Factory.new

    View Slide

  101. factory = MessagePack::Factory.new
    factory.register_type(
    0x03,
    Date,
    packer: ...,
    unpacker: ...
    )
    Extension Type

    View Slide

  102. factory = MessagePack::Factory.new
    factory.register_type(
    0x03,
    Date,
    packer: ...,
    unpacker: ...
    )
    type

    View Slide

  103. factory = MessagePack::Factory.new
    factory.register_type(
    0x03,
    Date,
    packer: ...,
    unpacker: ...
    )
    class
    (matches subclasses!)

    View Slide

  104. factory = MessagePack::Factory.new
    factory.register_type(
    0x03,
    Date,
    packer: ...,
    unpacker: ...
    )
    serializer/
    deserializer

    View Slide

  105. packer: ->(date) {
    },
    unpacker: ->(str) {
    }

    View Slide

  106. packer: ->(date) {
    [date.year,
    date.month,
    date.day].pack("s< C C")
    },
    unpacker: ->(str) {
    }
    Array#pack

    View Slide

  107. packer: ->(date) {
    [date.year,
    date.month,
    date.day].pack("s< C C")
    },
    unpacker: ->(str) {
    year, month, date =
    str.unpack("s< C C")
    Date.new(year, month, day)
    }
    String#unpack

    View Slide

  108. packer: ->(date) {
    [date.year,
    date.month,
    date.day].pack("s< C C")
    },
    unpacker: ->(str) {
    year, month, date =
    str.unpack("s< C C")
    Date.new(year, month, day)
    }

    View Slide

  109. date = Date.new(2022, 9, 9)
    #=> Fri, 09 Sep 2022

    View Slide

  110. date = Date.new(2022, 9, 9)
    encoded_date = factory.dump(date)
    #=> "\xD6\x03\xE6\a\t\t"

    View Slide

  111. date = Date.new(2022, 9, 9)
    encoded_date = factory.dump(date)
    #=> "\xD6\x03\xE6\a\t\t"
    d603 e607 0909
    type 3 2022 09 09

    View Slide

  112. date = Date.new(2022, 9, 9)
    encoded_date = factory.dump(date)
    factory.load(encoded_date)
    #=> Fri, 09 Sep 2022

    View Slide

  113. date = Date.new(2022, 9, 9)
    encoded_date = factory.dump(date)
    factory.load(encoded_date)
    factory.dump({
    "foo" => [Object.new]
    })

    View Slide

  114. date = Date.new(2022, 9, 9)
    encoded_date = factory.dump(date)
    factory.load(encoded_date)
    factory.dump({
    "foo" => [Object.new]
    })
    NoMethodError: undefined method
    `to_msgpack' for <#Object:0x...>

    View Slide

  115. let’s talk about
    what we did
    what we did

    View Slide

  116. Marshal
    MessagePack

    View Slide

  117. factory.dump

    View Slide

  118. MSGPACK
    XX
    factory.dump
    Version Byte Prefix

    View Slide

  119. MSGPACK
    XX
    rescue NoMethodError
    factory.dump

    View Slide

  120. MSGPACK
    XX
    rescue NoMethodError
    Marshal.dump
    MARSHAL
    0408
    factory.dump

    View Slide

  121. ...

    View Slide

  122. ...

    View Slide

  123. Symbol
    Time
    DateTime
    Date
    BigDecimal

    View Slide

  124. Symbol
    Time
    DateTime
    Date
    BigDecimal
    00

    View Slide

  125. Symbol
    Time
    DateTime
    Date
    BigDecimal
    ActiveRecord::Base
    00 01

    View Slide

  126. Symbol
    Time
    DateTime
    Date
    BigDecimal
    ActiveRecord::Base
    00 01
    attributes
    cached
    associations

    View Slide

  127. class Post
    has_many :comments
    end
    class Comment
    belongs_to :post,
    inverse_of: :comments
    end

    View Slide

  128. [
    [0, [
    [:comments, [[1, [:post, 0]],
    [2, [:post, 0]]]]]],
    [["Post", {"id"=>123, ...}],
    ["Comment", {"post_id"=>123, ..}],
    ["Comment", {"post_id"=>123, ..}]]
    ]
    associations
    records

    View Slide

  129. c801 2b06 9492 0091 92d7 0063 6f6d 6d65 6e74 7392 9201 9192 d600 706f 7374 0092 0291 92d6 0070 6f73 7400 92a4 506f 7374 84a2 6964 01a5 7469 746c 65b7 4361 6368 696e 6720 5769 7468 6f75 7420 4d61 7273
    6861 6caa 6372 6561 7465 645f 6174 c70f 08f8 cd68 6200 0000 0058 cccd 0d55 5443 aa75 7064 6174 6564 5f61 74c7 0f08 f8cd 6862 0000 0000 58cc cd0d 5554 4392 a743 6f6d 6d65 6e74 84a7 706f 7374 5f69 6401
    a269 6401 aa63 7265 6174 6564 5f61 74c7 0f08 2a79 7862 0000 0000 08b8 4630 5554 43aa 7570 6461 7465 645f 6174 c70f 082a 7978 6200 0000 0008 b846 3055 5443 92a7 436f 6d6d 656e 7484 a269 6402 aa63 7265
    6174 6564 5f61 74c7 0f08 7979 7862 0000 0000 b0c2 7c1f 5554 43aa 7570 6461 7465 645f 6174 c70f 0879 7978 6200 0000 00b0 c27c 1f55 5443 a770 6f73 745f 6964 01
    ActiveRecordCoder + MessagePack
    (303 bytes)

    View Slide

  130. c801 2b06 9492 0091 92d7 0063 6f6d 6d65 6e74 7392 9201 9192 d600 706f 7374 0092 0291 92d6 0070 6f73 7400 92a4 506f 7374 84a2 6964 01a5 7469 746c 65b7 4361 6368 696e 6720 5769 7468 6f75 7420 4d61 7273
    6861 6caa 6372 6561 7465 645f 6174 c70f 08f8 cd68 6200 0000 0058 cccd 0d55 5443 aa75 7064 6174 6564 5f61 74c7 0f08 f8cd 6862 0000 0000 58cc cd0d 5554 4392 a743 6f6d 6d65 6e74 84a7 706f 7374 5f69 6401
    a269 6401 aa63 7265 6174 6564 5f61 74c7 0f08 2a79 7862 0000 0000 08b8 4630 5554 43aa 7570 6461 7465 645f 6174 c70f 082a 7978 6200 0000 0008 b846 3055 5443 92a7 436f 6d6d 656e 7484 a269 6402 aa63 7265
    6174 6564 5f61 74c7 0f08 7979 7862 0000 0000 b0c2 7c1f 5554 43aa 7570 6461 7465 645f 6174 c70f 0879 7978 6200 0000 00b0 c27c 1f55 5443 a770 6f73 745f 6964 01
    0408 6f3a 0950 6f73 7411 3a10 406e 6577 5f72 6563 6f72 6446 3a10 4061 7474 7269 6275 7465 736f 3a22 4163 7469 7665 4d6f 6465 6c3a 3a4c 617a 7941 7474 7269 6275 7465 5365 740c 3b07 7b09 4922 0769 6406
    3a06 4554 6f3a 2941 6374 6976 654d 6f64 656c 3a3a 4174 7472 6962 7574 653a 3a46 726f 6d44 6174 6162 6173 650a 3a0a 406e 616d 6540 083a 1c40 7661 6c75 655f 6265 666f 7265 5f74 7970 655f 6361 7374 6906
    3a0a 4074 7970 656f 3a45 4163 7469 7665 5265 636f 7264 3a3a 436f 6e6e 6563 7469 6f6e 4164 6170 7465 7273 3a3a 5351 4c69 7465 3341 6461 7074 6572 3a3a 5351 4c69 7465 3349 6e74 6567 6572 093a 0f40 7072
    6563 6973 696f 6e30 3a0b 4073 6361 6c65 303a 0b40 6c69 6d69 7430 3a0b 4072 616e 6765 6f3a 0a52 616e 6765 083a 0965 7863 6c54 3a0a 6265 6769 6e6c 2d09 0000 0000 0000 0080 3a08 656e 646c 2b09 0000 0000
    0000 0080 3a18 406f 7269 6769 6e61 6c5f 6174 7472 6962 7574 6530 3a0b 4076 616c 7565 6906 4922 0a74 6974 6c65 063b 0954 6f3b 0a0a 3b0b 400e 3b0c 4922 1c43 6163 6869 6e67 2057 6974 686f 7574 204d 6172
    7368 616c 063b 0954 3b0d 6f3a 1e41 6374 6976 654d 6f64 656c 3a3a 5479 7065 3a3a 5374 7269 6e67 0a3a 0a40 7472 7565 4922 0674 063b 0954 3a0b 4066 616c 7365 4922 0666 063b 0954 3b0f 303b 1030 3b11 303b
    1730 3b18 4922 1c43 6163 6869 6e67 2057 6974 686f 7574 204d 6172 7368 616c 063b 0954 4922 0f63 7265 6174 6564 5f61 7406 3b09 546f 3b0a 0a3b 0b40 153b 0c49 221f 3230 3232 2d30 342d 3237 2030 353a 3030
    3a34 302e 3233 3135 3931 063b 0954 3b0d 553a 4a41 6374 6976 6552 6563 6f72 643a 3a41 7474 7269 6275 7465 4d65 7468 6f64 733a 3a54 696d 655a 6f6e 6543 6f6e 7665 7273 696f 6e3a 3a54 696d 655a 6f6e 6543
    6f6e 7665 7274 6572 5b09 3a0b 5f5f 7632 5f5f 5b00 5b00 6f3a 2141 6374 6976 6552 6563 6f72 643a 3a54 7970 653a 3a44 6174 6554 696d 6508 3b0f 690b 3b10 303b 1130 3b17 303b 1855 3a20 4163 7469 7665 5375
    7070 6f72 743a 3a54 696d 6557 6974 685a 6f6e 655b 0849 753a 0954 696d 650d 658f 1ec0 a788 8302 063a 097a 6f6e 6549 2208 5554 4306 3b09 4649 2208 5554 4306 3b09 5449 753b 200d 658f 1ec0 a788 8302 063b
    2140 1f49 220f 7570 6461 7465 645f 6174 063b 0954 6f3b 0a0a 3b0b 4023 3b0c 4922 1f32 3032 322d 3034 2d32 3720 3035 3a30 303a 3430 2e32 3331 3539 3106 3b09 543b 0d55 3b1c 5b09 3b1d 5b00 5b00 401c 3b17
    303b 1855 3b1f 5b08 4975 3b20 0d65 8f1e c0a7 8883 0206 3b21 401f 4021 4975 3b20 0d65 8f1e c0a7 8883 0206 3b21 401f 3a0c 4076 616c 7565 737b 0940 0869 0640 0e40 1040 1540 1740 2340 253a 0b40 7479 7065
    737d 0940 0840 0a40 0e40 1140 1540 1840 2340 266f 3a1d 4163 7469 7665 4d6f 6465 6c3a 3a54 7970 653a 3a56 616c 7565 083b 0f30 3b10 303b 1130 3a16 4061 6464 6974 696f 6e61 6c5f 7479 7065 737b 003a 1840
    6465 6661 756c 745f 6174 7472 6962 7574 6573 7b06 4008 6f3b 0a09 3b0b 4008 3b0c 303b 0d40 0a3b 1730 3a13 4063 6173 7465 645f 7661 6c75 6573 7b00 3a12 406d 6174 6572 6961 6c69 7a65 6454 3a17 4061 7373
    6f63 6961 7469 6f6e 5f63 6163 6865 7b06 3a0d 636f 6d6d 656e 7473 553a 3341 6374 6976 6552 6563 6f72 643a 3a41 7373 6f63 6961 7469 6f6e 733a 3a48 6173 4d61 6e79 4173 736f 6369 6174 696f 6e5b 073b 2a5b
    0f5b 073a 0b40 6f77 6e65 7240 005b 073a 1340 6469 7361 626c 655f 6a6f 696e 7346 5b07 3a0c 406c 6f61 6465 6454 5b07 3a0c 4074 6172 6765 745b 076f 3a0c 436f 6d6d 656e 7411 3b06 463b 076f 3b08 0c3b 077b
    0949 220c 706f 7374 5f69 6406 3b09 466f 3b0a 0a3b 0b40 413b 0c69 063b 0d40 0a3b 1730 3b18 6906 4008 6f3b 0a0a 3b0b 4008 3b0c 6906 3b0d 400a 3b17 303b 1869 0640 156f 3b0a 0a3b 0b40 153b 0c49 221f 3230
    3232 2d30 352d 3039 2030 323a 3135 3a30 362e 3830 3939 3431 063b 0954 3b0d 553b 1c5b 093b 1d5b 005b 0040 1c3b 1730 3b18 553b 1f5b 0849 753b 200d 2291 1ec0 d55b 6c3c 063b 2140 1f40 2149 753b 200d 2291
    1ec0 d55b 6c3c 063b 2140 1f40 236f 3b0a 0a3b 0b40 233b 0c49 221f 3230 3232 2d30 352d 3039 2030 323a 3135 3a30 362e 3830 3939 3431 063b 0954 3b0d 553b 1c5b 093b 1d5b 005b 0040 1c3b 1730 3b18 553b 1f5b
    0849 753b 200d 2291 1ec0 d55b 6c3c 063b 2140 1f40 2149 753b 200d 2291 1ec0 d55b 6c3c 063b 2140 1f3b 227b 0940 0869 0640 1540 4540 2340 4f49 220c 706f 7374 5f69 6406 3b09 5469 063b 237d 0940 0840 0a40
    1540 4640 2340 5040 5940 0a40 303b 257b 003b 267b 0640 086f 3b0a 093b 0b40 083b 0c30 3b0d 400a 3b17 303b 277b 003b 2854 3b29 7b06 3a09 706f 7374 553a 3541 6374 6976 6552 6563 6f72 643a 3a41 7373 6f63
    6961 7469 6f6e 733a 3a42 656c 6f6e 6773 546f 4173 736f 6369 6174 696f 6e5b 073b 315b 0c5b 073b 2c40 3e5b 073b 2d46 5b07 3b2e 545b 073b 2f40 005b 073a 1140 7374 616c 655f 7374 6174 6549 2206 3106 3b09
    465b 073a 0d40 7570 6461 7465 6446 5b07 3a17 4061 7373 6f63 6961 7469 6f6e 5f73 636f 7065 303a 0e40 7265 6164 6f6e 6c79 463a 1b40 7072 6576 696f 7573 6c79 5f6e 6577 5f72 6563 6f72 6446 3a0f 4064 6573
    7472 6f79 6564 463a 1c40 6d61 726b 6564 5f66 6f72 5f64 6573 7472 7563 7469 6f6e 463a 1e40 6465 7374 726f 7965 645f 6279 5f61 7373 6f63 6961 7469 6f6e 303a 1e40 5f73 7461 7274 5f74 7261 6e73 6163 7469
    6f6e 5f73 7461 7465 303a 1140 7072 696d 6172 795f 6b65 7940 083a 1440 7374 7269 6374 5f6c 6f61 6469 6e67 463a 1940 7374 7269 6374 5f6c 6f61 6469 6e67 5f6d 6f64 653a 0861 6c6c 6f3b 301b 3b06 463b 076f
    3a1e 4163 7469 7665 4d6f 6465 6c3a 3a41 7474 7269 6275 7465 5365 7406 3b07 7b09 4008 6f3b 0a0a 3b0b 4008 3b0c 6907 3b0d 400a 3b17 303b 1869 0740 156f 3b0a 0a3b 0b40 153b 0c55 3b1f 5b08 4975 3b20 0d22
    911e c08e 0f98 4106 3b21 401f 4021 4975 3b20 0d22 911e c08e 0f98 4106 3b21 401f 3b0d 4046 3b17 303b 1840 7040 236f 3b0a 0a3b 0b40 233b 0c55 3b1f 5b08 4072 4021 4975 3b20 0d22 911e c08e 0f98 4106 3b21
    401f 3b0d 4050 3b17 303b 1840 7540 596f 3b0a 0a3b 0b40 593b 0c69 063b 0d40 0a3b 1730 3b18 6906 3b29 7b06 3b31 553b 325b 073b 315b 0c5b 073b 2c40 6b5b 073b 2d46 5b07 3b2e 545b 073b 2f40 005b 073b 3349
    2206 3106 3b09 465b 073b 3446 5b07 3b35 303b 3646 3b37 543b 3846 3b39 463b 3a30 3b3b 303b 3c40 083b 3d46 3b3e 3b3f 3a1d 406d 7574 6174 696f 6e73 5f66 726f 6d5f 6461 7461 6261 7365 303a 2440 5f6e 6577
    5f72 6563 6f72 645f 6265 666f 7265 5f6c 6173 745f 636f 6d6d 6974 543a 1840 7661 6c69 6461 7469 6f6e 5f63 6f6e 7465 7874 303a 0c40 6572 726f 7273 6f3a 1841 6374 6976 654d 6f64 656c 3a3a 4572 726f 7273
    073a 0a40 6261 7365 406b 3b44 5b00 3a13 405f 746f 7563 685f 7265 636f 7264 543a 1540 5f61 6c72 6561 6479 5f63 616c 6c65 647b 063a 2961 7574 6f73 6176 655f 6173 736f 6369 6174 6564 5f72 6563 6f72 6473
    5f66 6f72 5f70 6f73 7446 3a20 406d 7574 6174 696f 6e73 5f62 6566 6f72 655f 6c61 7374 5f73 6176 656f 3a2a 4163 7469 7665 4d6f 6465 6c3a 3a41 7474 7269 6275 7465 4d75 7461 7469 6f6e 5472 6163 6b65 7207
    3b07 6f3b 4006 3b07 7b09 4008 6f3a 2541 6374 6976 654d 6f64 656c 3a3a 4174 7472 6962 7574 653a 3a46 726f 6d55 7365 720a 3b0b 4008 3b0c 6907 3b0d 400a 3b17 6f3b 0a0a 3b0b 4008 3b0c 303b 0d40 0a3b 1730
    3b18 303b 1869 0740 156f 3b4c 0a3b 0b40 153b 0c40 723b 0d40 463b 176f 3b0a 0a3b 0b40 153b 0c30 3b0d 4046 3b17 303b 1830 3b18 4070 4023 6f3b 4c0a 3b0b 4023 3b0c 4072 3b0d 4050 3b17 6f3b 0a0a 3b0b 4023
    3b0c 303b 0d40 503b 1730 3b18 303b 1840 7540 596f 3b4c 0a3b 0b40 593b 0c69 063b 0d40 0a3b 176f 3b0a 093b 0b40 593b 0c30 3b0d 400a 3b17 303b 1869 063a 1440 666f 7263 6564 5f63 6861 6e67 6573 7b00 3a1f
    405f 636f 6d6d 6974 7465 645f 616c 7265 6164 795f 6361 6c6c 6564 463a 1f40 5f74 7269 6767 6572 5f64 6573 7472 6f79 5f63 616c 6c62 6163 6b46 3a1e 405f 7472 6967 6765 725f 7570 6461 7465 5f63 616c 6c62
    6163 6b46 5b07 3b33 305b 073a 1f40 7265 706c 6163 6564 5f6f 725f 6164 6465 645f 7461 7267 6574 736f 3a08 5365 7406 3a0a 4068 6173 687d 0046 5b07 3a15 4061 7373 6f63 6961 7469 6f6e 5f69 6473 305b 073b
    356f 3a23 436f 6d6d 656e 743a 3a41 6374 6976 6552 6563 6f72 645f 5265 6c61 7469 6f6e 0d3a 0b40 6b6c 6173 7363 0c43 6f6d 6d65 6e74 3a0b 4074 6162 6c65 6f3a 1041 7265 6c3a 3a54 6162 6c65 093b 0b49 220d
    636f 6d6d 656e 7473 063b 0954 3b56 4001 963a 1140 7479 7065 5f63 6173 7465 726f 3a22 4163 7469 7665 5265 636f 7264 3a3a 5479 7065 4361 7374 6572 3a3a 4d61 7006 3b56 4001 963a 1140 7461 626c 655f 616c
    6961 7330 3b22 7b07 3a0e 6578 7465 6e64 696e 675b 003a 0a77 6865 7265 6f3a 2841 6374 6976 6552 6563 6f72 643a 3a52 656c 6174 696f 6e3a 3a57 6865 7265 436c 6175 7365 063a 1040 7072 6564 6963 6174 6573
    5b06 6f3a 1a41 7265 6c3a 3a4e 6f64 6573 3a3a 4571 7561 6c69 7479 073a 0a40 6c65 6674 533a 2041 7265 6c3a 3a41 7474 7269 6275 7465 733a 3a41 7474 7269 6275 7465 073a 0d72 656c 6174 696f 6e40 0197 3a09
    6e61 6d65 4041 3a0b 4072 6967 6874 6f3a 2b41 6374 6976 6552 6563 6f72 643a 3a52 656c 6174 696f 6e3a 3a51 7565 7279 4174 7472 6962 7574 6509 3b0b 4041 3b0c 6906 3b0d 400a 3b17 303b 2e46 3a17 4070 7265
    6469 6361 7465 5f62 7569 6c64 6572 6f3a 2341 6374 6976 6552 6563 6f72 643a 3a50 7265 6469 6361 7465 4275 696c 6465 7207 3b57 6f3a 2041 6374 6976 6552 6563 6f72 643a 3a54 6162 6c65 4d65 7461 6461 7461
    083b 5640 0196 3a10 4061 7265 6c5f 7461 626c 6540 0197 3a10 4072 6566 6c65 6374 696f 6e30 3a0e 4068 616e 646c 6572 735b 0a5b 0763 0853 6574 6f3a 3141 6374 6976 6552 6563 6f72 643a 3a50 7265 6469 6361
    7465 4275 696c 6465 723a 3a41 7272 6179 4861 6e64 6c65 7206 3b67 4001 a15b 0763 0a41 7272 6179 6f3b 6d06 3b67 4001 a15b 0763 1b41 6374 6976 6552 6563 6f72 643a 3a52 656c 6174 696f 6e6f 3a34 4163 7469
    7665 5265 636f 7264 3a3a 5072 6564 6963 6174 6542 7569 6c64 6572 3a3a 5265 6c61 7469 6f6e 4861 6e64 6c65 7200 5b07 630a 5261 6e67 656f 3a31 4163 7469 7665 5265 636f 7264 3a3a 5072 6564 6963 6174 6542
    7569 6c64 6572 3a3a 5261 6e67 6548 616e 646c 6572 063b 6740 01a1 5b07 6310 4261 7369 634f 626a 6563 746f 3a37 4163 7469 7665 5265 636f 7264 3a3a 5072 6564 6963 6174 6542 7569 6c64 6572 3a3a 4261 7369
    634f 626a 6563 7448 616e 646c 6572 063b 6740 01a1 3a17 4064 656c 6567 6174 655f 746f 5f6b 6c61 7373 463a 1340 6675 7475 7265 5f72 6573 756c 7430 3a0d 4072 6563 6f72 6473 305b 073a 0b40 7072 6f78 796f
    3a37 436f 6d6d 656e 743a 3a41 6374 6976 6552 6563 6f72 645f 4173 736f 6369 6174 696f 6e73 5f43 6f6c 6c65 6374 696f 6e50 726f 7879 113a 1140 6173 736f 6369 6174 696f 6e40 363b 5640 0196 3b57 4001 973b
    227b 003b 2e46 3b67 4001 a13b 7146 3b72 303b 7330 3a0a 4074 616b 6530 3a0d 406f 6666 7365 7473 303a 0b40 7363 6f70 6530 5b07 3a11 405f 7761 735f 6c6f 6164 6564 303b 3646 3b37 463b 3846 3b39 463b 3a30
    3b3b 303b 3c40 083b 3d46 3b3e 3b3f
    Marshal
    (4014 bytes)

    View Slide

  131. Rails cache memcached fill percent

    View Slide

  132. View Slide

  133. Symbol
    Time
    DateTime
    Date
    BigDecimal
    ActiveRecord::Base
    00 01 02
    HashWithIndifferentAccess

    View Slide

  134. Symbol
    Time
    DateTime
    Date
    BigDecimal
    ActiveRecord::Base
    00 01 02
    HashWithIndifferentAccess < Hash
    core type subclass

    View Slide

  135. hwia = {}.with_indifferent_access

    View Slide

  136. hwia = {}.with_indifferent_access
    hwia.class
    #=> HashWithIndifferentAccess

    View Slide

  137. hwia = {}.with_indifferent_access
    hwia.class
    #=> HashWithIndifferentAccess
    encoded = MessagePack.pack(hwia)
    decoded = MessagePack.unpack(encoded)

    View Slide

  138. hwia = {}.with_indifferent_access
    hwia.class
    #=> HashWithIndifferentAccess
    encoded = MessagePack.pack(hwia)
    decoded = MessagePack.unpack(encoded)
    decoded.class
    #=> Hash

    View Slide

  139. /* ext/msgpack/packer.c */
    void msgpack_packer_write_value(msgpack_packer_t* pk, VALUE v)
    {
    switch(rb_type(v)) {
    case T_NIL:
    msgpack_packer_write_nil(pk);
    break;
    case T_HASH:
    msgpack_packer_write_hash_value(pk, v);
    break;
    default:
    msgpack_packer_write_other_value(pk, v);
    }
    }
    ... ...
    matches subclasses!
    v1.4.2

    View Slide

  140. factory = MessagePack::Factory.new

    View Slide

  141. factory = MessagePack::Factory.new
    factory.register_type(
    0x7,
    HashWithIndifferentAccess,
    packer: -> (value) {
    },
    unpacker: -> (str) {
    }
    )

    View Slide

  142. factory = MessagePack::Factory.new
    factory.register_type(
    0x7,
    HashWithIndifferentAccess,
    packer: -> (value) {
    factory.dump(value.to_h)
    },
    unpacker: -> (str) {
    }
    )

    View Slide

  143. factory = MessagePack::Factory.new
    factory.register_type(
    0x7,
    HashWithIndifferentAccess,
    packer: -> (value) {
    factory.dump(value.to_h)
    },
    unpacker: -> (str) {
    factory.load(value).
    with_indifferent_access
    }
    )

    View Slide

  144. /* ext/msgpack/packer.c */
    void msgpack_packer_write_value(msgpack_packer_t* pk, VALUE v)
    {
    switch(rb_type(v)) {
    case T_NIL:
    msgpack_packer_write_nil(pk);
    break;
    case T_HASH:
    msgpack_packer_write_hash_value(pk, v);
    break;
    default:
    msgpack_packer_write_other_value(pk, v);
    }
    }
    ... ...
    extension type
    v1.4.2

    View Slide

  145. /* ext/msgpack/packer.c */
    void msgpack_packer_write_value(msgpack_packer_t* pk, VALUE v)
    {
    switch(rb_type(v)) {
    case T_NIL:
    msgpack_packer_write_nil(pk);
    break;
    case T_HASH:
    msgpack_packer_write_hash_value(pk, v);
    break;
    default:
    msgpack_packer_write_other_value(pk, v);
    }
    }
    ... ...
    this matches first
    v1.4.2

    View Slide

  146. msgpack 1.4.3 (Jan, 2022)

    View Slide

  147. Symbol
    Time
    DateTime
    Date
    BigDecimal
    ActiveRecord::Base
    00 01 02 03
    HashWithIndifferentAccess
    ActiveSupport::TimeWithZone
    04 ...

    View Slide

  148. View Slide

  149. Symbol
    Time
    DateTime
    Date
    BigDecimal
    ActiveRecord::Base
    00 01 02 03 7f
    Object
    HashWithIndifferentAccess
    ActiveSupport::TimeWithZone
    04 ...

    View Slide

  150. class Task
    def initialize(title, content)
    @title, @content = title, content
    end
    def as_pack
    [@title, @content]
    end
    def self.from_pack(payload)
    title, content = payload
    new(title, content)
    end
    end
    “packable”

    View Slide

  151. MyStruct = Struct.new(:foo, :bar)
    MyStruct.include(Paquito::Struct)

    View Slide

  152. MyStruct = Struct.new(:foo, :bar)
    MyStruct.include(Paquito::Struct)
    my_struct = MyStruct.new("foo", "bar")
    my_struct.as_pack
    #=> [26450, "foo", "bar"]
    digest

    View Slide

  153. MyStruct = Struct.new(:foo, :bar)
    MyStruct.include(Paquito::Struct)
    my_struct = MyStruct.new("foo", "bar")
    my_struct.as_pack
    #=> [26450, "foo", "bar"]
    MyStruct.from_pack([26450, "foo", "bar"])
    #=> #

    View Slide

  154. MyStruct = Struct.new(:foo, :bar)
    MyStruct.include(Paquito::Struct)
    my_struct = MyStruct.new("foo", "bar")
    my_struct.as_pack
    #=> [26450, "foo", "bar"]
    MyStruct.from_pack([26450, "foo", "bar"])
    #=> #
    MyStruct.from_pack([123, "foo", "bar"])
    # MyStruct digests do not match!

    View Slide

  155. View Slide

  156. MessagePack

    View Slide

  157. Shopify/paquito

    View Slide

  158. Shopify/paquito
    Jean Boussier
    (@byroot)

    View Slide

  159. Chris Salzberg
    @shioyama
    Thanks!

    View Slide