Caching With MessagePack

RubyKaigi 2022 talk about how we replaced Marshal with MessagePack in the Rails cache of the Shopify monolith.

Transcript

1. CACHING
   MSGPACK
   Chris Salzberg
   with
   with
   

   View Slide

2. Chris Salzberg
   @shioyama
   Staff Dev @ Ruby & Rails Infra
   Team: Rails in Core
   

   View Slide

3. View Slide

4. View Slide

5. before After
   

   View Slide

6. before After
   cache
   

   View Slide

7. let’s talk about
   caching
   caching
   

   View Slide

8. Application
   

   View Slide

9. Application
   actions
   

   View Slide

10. Application
    actions pages
    

    View Slide

11. Application
    actions pages fragments
    

    View Slide

12. Application
    actions pages fragments queries
    

    View Slide

13. Application
    actions pages fragments queries responses
    

    View Slide

14. Application
    actions pages fragments queries responses
    file redis memcached
    memory
    

    View Slide

15. Application
    actions pages fragments queries responses
    file redis memcached
    memory
    cache
    

    View Slide

16. Application
    actions pages fragments queries responses
    file redis memcached
    memory
    cache cache
    

    View Slide

17. Application
    actions pages fragments queries responses
    file redis memcached
    memory
    cache cache cache ...
    

    View Slide

18. View Slide

19. Application
    actions pages fragments queries responses
    

    View Slide

20. Application
    actions pages fragments queries responses
    Cache
    

    View Slide

21. Application
    actions pages fragments queries
    file redis memcached
    memory
    Cache
    responses
    

    View Slide

22. file redis memcached
    memory
    Application
    actions pages fragments queries
    Cache
    responses
    Marshal
    

    View Slide

23. View Slide

24. irb(main):010:0> Marshal.method(:dump).source
    

    View Slide

25. irb(main):010:0> Marshal.method(:dump).source
    Could not locate source for dump!
    (MethodSource::SourceNotFoundError)
    

    View Slide

26. let’s talk about
    Marshal
    Marshal
    

    View Slide

27. class Post < ApplicationRecord
    end
    

    View Slide

28. post = Post.create(title: "Caching With MessagePack")
    #=> #

    View Slide

29. post = Post.create(title: "Caching With MessagePack")
    #=> #serialized_post = Marshal.dump(post)
    

    View Slide

30. 1607 bytes!!!
    post = Post.create(title: "Caching With MessagePack")
    #=> #serialized_post = Marshal.dump(post)
    #=> "\x04\bo:\tPost\x1A:\x10@new_recordF:\x10@attributeso:\x1EActiveModel::AttributeSet\x06;\a{\tI\"\aid\x06:\x06ETo:)ActiveMod
    el::Attribute::FromDatabase\n:\n@name@\b:\x1C@value_before_type_casti\x06:\n@typeo:EActiveRecord::ConnectionAdapters::SQLite3Ad
    apter::SQLite3Integer\t:\x0F@precision0:\v@scale0:\v@limit0:\v@rangeo:\nRange\b:\texclT:\nbeginl-\t\x00\x00\x00\x00\x00\x00\x00
    \x80:\bendl+\t\x00\x00\x00\x00\x00\x00\x00\x80:\x18@original_attribute0:\v@valuei\x06I\"\ntitle\x06;\tTo;\n\n;\v@\x0E;\fI\"\x1C
    Caching With MessagePack\x06;\tT;\ro:\x1EActiveModel::Type::String\n:\n@trueI\"\x06t\x06;\tT:\v@falseI\"\x06f\x06;\tT;\x0F0;\x1
    00;\x110;\x170;\x18I\"\x1CCaching With MessagePack\x06;\tTI\"\x0Fcreated_at\x06;\tTo;\n\n;\v@\x15;\fU: ActiveSupport::TimeWithZ
    one[\bIu:\tTime\re\x8F\x1E\xC0\xA7\x88\x83\x02\x06:\tzoneI\"\bUTC\x06;\tFI\"\bUTC\x06;\tTIu;\x1D\re\x8F\x1E\xC0\xA7\x88\x83\x02
    \x06;\x1E@\x19;\rU:JActiveRecord::AttributeMethods::TimeZoneConversion::TimeZoneConverter[\t:\v__v2__[\x00[\x00o:!ActiveRecord:
    :Type::DateTime\b;\x0Fi\v;\x100;\x110;\x170;\x18@\x17I\"\x0Fupdated_at\x06;\tTo;\n\n;\v@\";\fU;\x1C[\b@\x1A@\eIu;\x1D\re\x8F\x1
    E\xC0\xA7\x88\x83\x02\x06;\x1E@\x19;\rU;\x1F[\t; [\x00[\x00@!;\x170;\x18@$:\x17@association_cache{\x00:\x0E@readonlyF:\e@previo
    usly_new_recordT:\x0F@destroyedF:\x1C@marked_for_destructionF:\x1E@destroyed_by_association0:\x1E@_start_transaction_state0:\x1
    1@primary_key@\b:\x14@strict_loadingF:\x19@strict_loading_mode:\ball:$@_new_record_before_last_commitT:\x18@validation_context0
    :\f@errorso:\x18ActiveModel::Errors\a:\n@base@\x00;/[\x00:\x13@_touch_recordT:\x1D@mutations_from_database0: @mutations_before_
    last_saveo:*ActiveModel::AttributeMutationTracker\x06;\ao;\b\x06;\a{\t@\bo:%ActiveModel::Attribute::FromUser\n;\v@\b;\fi\x06;\r
    @\n;\x17o;\n\n;\v@\b;\f0;\r@\n;\x170;\x180;\x18i\x06@\x0Eo;6\n;\v@\x0E;\fI\"\x1CCaching With MessagePack\x06;\tT;\r@\x11;\x17o;
    \n\t;\v@\x0E;\f0;\r@\x11;\x170;\x18@\x10@\x15o;6\n;\v@\x15;\f@\x1A;\r@\x1D;\x17o;\n\n;\v@\x15;\f0;\r@\x1D;\x170;\x180;\x18@\x17
    @\"o;6\n;\v@\";\f@\x1A;\r@';\x17o;\n\n;\v@\";\f0;\r@';\x170;\x180;\x18@$:\x1F@_committed_already_calledF:\x1F@_trigger_destroy_
    callbackF:\x1E@_trigger_update_callbackF"
    

    View Slide

31. post = Post.create(title: "Caching With MessagePack")
    #=> #serialized_post = Marshal.dump(post)
    #=> "\x04\bo:\tPost\x1A:\x10@new_recordF:\x10@attributeso:\x1EActiveModel::AttributeSet\x06;\a{\tI\"\aid\x06:\x06ETo:)ActiveMod
    el::Attribute::FromDatabase\n:\n@name@\b:\x1C@value_before_type_casti\x06:\n@typeo:EActiveRecord::ConnectionAdapters::SQLite3Ad
    apter::SQLite3Integer\t:\x0F@precision0:\v@scale0:\v@limit0:\v@rangeo:\nRange\b:\texclT:\nbeginl-\t\x00\x00\x00\x00\x00\x00\x00
    \x80:\bendl+\t\x00\x00\x00\x00\x00\x00\x00\x80:\x18@original_attribute0:\v@valuei\x06I\"\ntitle\x06;\tTo;\n\n;\v@\x0E;\fI\"\x1C
    Caching With MessagePack\x06;\tT;\ro:\x1EActiveModel::Type::String\n:\n@trueI\"\x06t\x06;\tT:\v@falseI\"\x06f\x06;\tT;\x0F0;\x1
    00;\x110;\x170;\x18I\"\x1CCaching With MessagePack\x06;\tTI\"\x0Fcreated_at\x06;\tTo;\n\n;\v@\x15;\fU: ActiveSupport::TimeWithZ
    one[\bIu:\tTime\re\x8F\x1E\xC0\xA7\x88\x83\x02\x06:\tzoneI\"\bUTC\x06;\tFI\"\bUTC\x06;\tTIu;\x1D\re\x8F\x1E\xC0\xA7\x88\x83\x02
    \x06;\x1E@\x19;\rU:JActiveRecord::AttributeMethods::TimeZoneConversion::TimeZoneConverter[\t:\v__v2__[\x00[\x00o:!ActiveRecord:
    :Type::DateTime\b;\x0Fi\v;\x100;\x110;\x170;\x18@\x17I\"\x0Fupdated_at\x06;\tTo;\n\n;\v@\";\fU;\x1C[\b@\x1A@\eIu;\x1D\re\x8F\x1
    E\xC0\xA7\x88\x83\x02\x06;\x1E@\x19;\rU;\x1F[\t; [\x00[\x00@!;\x170;\x18@$:\x17@association_cache{\x00:\x0E@readonlyF:\e@previo
    usly_new_recordT:\x0F@destroyedF:\x1C@marked_for_destructionF:\x1E@destroyed_by_association0:\x1E@_start_transaction_state0:\x1
    1@primary_key@\b:\x14@strict_loadingF:\x19@strict_loading_mode:\ball:$@_new_record_before_last_commitT:\x18@validation_context0
    :\f@errorso:\x18ActiveModel::Errors\a:\n@base@\x00;/[\x00:\x13@_touch_recordT:\x1D@mutations_from_database0: @mutations_before_
    last_saveo:*ActiveModel::AttributeMutationTracker\x06;\ao;\b\x06;\a{\t@\bo:%ActiveModel::Attribute::FromUser\n;\v@\b;\fi\x06;\r
    @\n;\x17o;\n\n;\v@\b;\f0;\r@\n;\x170;\x180;\x18i\x06@\x0Eo;6\n;\v@\x0E;\fI\"\x1CCaching With MessagePack\x06;\tT;\r@\x11;\x17o;
    \n\t;\v@\x0E;\f0;\r@\x11;\x170;\x18@\x10@\x15o;6\n;\v@\x15;\f@\x1A;\r@\x1D;\x17o;\n\n;\v@\x15;\f0;\r@\x1D;\x170;\x180;\x18@\x17
    @\"o;6\n;\v@\";\f@\x1A;\r@';\x17o;\n\n;\v@\";\f0;\r@';\x170;\x180;\x18@$:\x1F@_committed_already_calledF:\x1F@_trigger_destroy_
    callbackF:\x1E@_trigger_update_callbackF"
    Constants
    

    View Slide

32. post = Post.create(title: "Caching With MessagePack")
    #=> #serialized_post = Marshal.dump(post)
    #=> "\x04\bo:\tPost\x1A:\x10@new_recordF:\x10@attributeso:\x1EActiveModel::AttributeSet\x06;\a{\tI\"\aid\x06:\x06ETo:)ActiveMod
    el::Attribute::FromDatabase\n:\n@name@\b:\x1C@value_before_type_casti\x06:\n@typeo:EActiveRecord::ConnectionAdapters::SQLite3Ad
    apter::SQLite3Integer\t:\x0F@precision0:\v@scale0:\v@limit0:\v@rangeo:\nRange\b:\texclT:\nbeginl-\t\x00\x00\x00\x00\x00\x00\x00
    \x80:\bendl+\t\x00\x00\x00\x00\x00\x00\x00\x80:\x18@original_attribute0:\v@valuei\x06I\"\ntitle\x06;\tTo;\n\n;\v@\x0E;\fI\"\x1C
    Caching With MessagePack\x06;\tT;\ro:\x1EActiveModel::Type::String\n:\n@trueI\"\x06t\x06;\tT:\v@falseI\"\x06f\x06;\tT;\x0F0;\x1
    00;\x110;\x170;\x18I\"\x1CCaching With MessagePack\x06;\tTI\"\x0Fcreated_at\x06;\tTo;\n\n;\v@\x15;\fU: ActiveSupport::TimeWithZ
    one[\bIu:\tTime\re\x8F\x1E\xC0\xA7\x88\x83\x02\x06:\tzoneI\"\bUTC\x06;\tFI\"\bUTC\x06;\tTIu;\x1D\re\x8F\x1E\xC0\xA7\x88\x83\x02
    \x06;\x1E@\x19;\rU:JActiveRecord::AttributeMethods::TimeZoneConversion::TimeZoneConverter[\t:\v__v2__[\x00[\x00o:!ActiveRecord:
    :Type::DateTime\b;\x0Fi\v;\x100;\x110;\x170;\x18@\x17I\"\x0Fupdated_at\x06;\tTo;\n\n;\v@\";\fU;\x1C[\b@\x1A@\eIu;\x1D\re\x8F\x1
    E\xC0\xA7\x88\x83\x02\x06;\x1E@\x19;\rU;\x1F[\t; [\x00[\x00@!;\x170;\x18@$:\x17@association_cache{\x00:\x0E@readonlyF:\e@previo
    usly_new_recordT:\x0F@destroyedF:\x1C@marked_for_destructionF:\x1E@destroyed_by_association0:\x1E@_start_transaction_state0:\x1
    1@primary_key@\b:\x14@strict_loadingF:\x19@strict_loading_mode:\ball:$@_new_record_before_last_commitT:\x18@validation_context0
    :\f@errorso:\x18ActiveModel::Errors\a:\n@base@\x00;/[\x00:\x13@_touch_recordT:\x1D@mutations_from_database0: @mutations_before_
    last_saveo:*ActiveModel::AttributeMutationTracker\x06;\ao;\b\x06;\a{\t@\bo:%ActiveModel::Attribute::FromUser\n;\v@\b;\fi\x06;\r
    @\n;\x17o;\n\n;\v@\b;\f0;\r@\n;\x170;\x180;\x18i\x06@\x0Eo;6\n;\v@\x0E;\fI\"\x1CCaching With MessagePack\x06;\tT;\r@\x11;\x17o;
    \n\t;\v@\x0E;\f0;\r@\x11;\x170;\x18@\x10@\x15o;6\n;\v@\x15;\f@\x1A;\r@\x1D;\x17o;\n\n;\v@\x15;\f0;\r@\x1D;\x170;\x180;\x18@\x17
    @\"o;6\n;\v@\";\f@\x1A;\r@';\x17o;\n\n;\v@\";\f0;\r@';\x170;\x180;\x18@$:\x1F@_committed_already_calledF:\x1F@_trigger_destroy_
    callbackF:\x1E@_trigger_update_callbackF"
    ivars
    

    View Slide

33. post = Post.create(title: "Caching With MessagePack")
    #=> #serialized_post = Marshal.dump(post)
    #=> "\x04\bo:\tPost\x1A:\x10@new_recordF:\x10@attributeso:\x1EActiveModel::AttributeSet\x06;\a{\tI\"\aid\x06:\x06ETo:)ActiveMod
    el::Attribute::FromDatabase\n:\n@name@\b:\x1C@value_before_type_casti\x06:\n@typeo:EActiveRecord::ConnectionAdapters::SQLite3Ad
    apter::SQLite3Integer\t:\x0F@precision0:\v@scale0:\v@limit0:\v@rangeo:\nRange\b:\texclT:\nbeginl-\t\x00\x00\x00\x00\x00\x00\x00
    \x80:\bendl+\t\x00\x00\x00\x00\x00\x00\x00\x80:\x18@original_attribute0:\v@valuei\x06I\"\ntitle\x06;\tTo;\n\n;\v@\x0E;\fI\"\x1C
    Caching With MessagePack\x06;\tT;\ro:\x1EActiveModel::Type::String\n:\n@trueI\"\x06t\x06;\tT:\v@falseI\"\x06f\x06;\tT;\x0F0;\x1
    00;\x110;\x170;\x18I\"\x1CCaching With MessagePack\x06;\tTI\"\x0Fcreated_at\x06;\tTo;\n\n;\v@\x15;\fU: ActiveSupport::TimeWithZ
    one[\bIu:\tTime\re\x8F\x1E\xC0\xA7\x88\x83\x02\x06:\tzoneI\"\bUTC\x06;\tFI\"\bUTC\x06;\tTIu;\x1D\re\x8F\x1E\xC0\xA7\x88\x83\x02
    \x06;\x1E@\x19;\rU:JActiveRecord::AttributeMethods::TimeZoneConversion::TimeZoneConverter[\t:\v__v2__[\x00[\x00o:!ActiveRecord:
    :Type::DateTime\b;\x0Fi\v;\x100;\x110;\x170;\x18@\x17I\"\x0Fupdated_at\x06;\tTo;\n\n;\v@\";\fU;\x1C[\b@\x1A@\eIu;\x1D\re\x8F\x1
    E\xC0\xA7\x88\x83\x02\x06;\x1E@\x19;\rU;\x1F[\t; [\x00[\x00@!;\x170;\x18@$:\x17@association_cache{\x00:\x0E@readonlyF:\e@previo
    usly_new_recordT:\x0F@destroyedF:\x1C@marked_for_destructionF:\x1E@destroyed_by_association0:\x1E@_start_transaction_state0:\x1
    1@primary_key@\b:\x14@strict_loadingF:\x19@strict_loading_mode:\ball:$@_new_record_before_last_commitT:\x18@validation_context0
    :\f@errorso:\x18ActiveModel::Errors\a:\n@base@\x00;/[\x00:\x13@_touch_recordT:\x1D@mutations_from_database0: @mutations_before_
    last_saveo:*ActiveModel::AttributeMutationTracker\x06;\ao;\b\x06;\a{\t@\bo:%ActiveModel::Attribute::FromUser\n;\v@\b;\fi\x06;\r
    @\n;\x17o;\n\n;\v@\b;\f0;\r@\n;\x170;\x180;\x18i\x06@\x0Eo;6\n;\v@\x0E;\fI\"\x1CCaching With MessagePack\x06;\tT;\r@\x11;\x17o;
    \n\t;\v@\x0E;\f0;\r@\x11;\x170;\x18@\x10@\x15o;6\n;\v@\x15;\f@\x1A;\r@\x1D;\x17o;\n\n;\v@\x15;\f0;\r@\x1D;\x170;\x180;\x18@\x17
    @\"o;6\n;\v@\";\f@\x1A;\r@';\x17o;\n\n;\v@\";\f0;\r@';\x170;\x180;\x18@$:\x1F@_committed_already_calledF:\x1F@_trigger_destroy_
    callbackF:\x1E@_trigger_update_callbackF"
    Values
    

    View Slide

34. post = Post.create(title: "Caching With MessagePack")
    #=> #serialized_post = Marshal.dump(post)
    Marshal.load(serialized_post)
    #=> ## id: 1,
    # title: "Caching With MessagePack",
    # created_at: Wed, 27 Apr 2022 05:00:40.231591000 UTC +00:00,
    # updated_at: Wed, 27 Apr 2022 05:00:40.231591000 UTC +00:00>
    

    View Slide

35. marshal encodes
    the universe
    the universe
    

    View Slide

36. marshal.c
    #define MARSHAL_MAJOR 4
    #define MARSHAL_MINOR 8
    #define TYPE_NIL '0'
    #define TYPE_TRUE 'T'
    #define TYPE_FALSE 'F'
    #define TYPE_FIXNUM 'i'
    #define TYPE_BIGNUM 'l'
    #define TYPE_FLOAT 'f'
    #define TYPE_SYMBOL ':'
    #define TYPE_CLASS 'c'
    #define TYPE_MODULE 'm'
    #define TYPE_LINK '@'
    #define TYPE_SYMLINK ';'
    #define TYPE_STRING '"'
    #define TYPE_REGEXP '/'
    #define TYPE_ARRAY '['
    #define TYPE_HASH '{'
    #define TYPE_HASH_DEF '}'
    #define TYPE_STRUCT 'S'
    #define TYPE_OBJECT 'o'
    #define TYPE_IVAR 'I'
    #define TYPE_UCLASS 'C'
    #define TYPE_EXTENDED 'e'
    #define TYPE_USERDEF 'u'
    #define TYPE_USRMARSHAL 'U'
    

    View Slide

37. marshal.c
    #define MARSHAL_MAJOR 4
    #define MARSHAL_MINOR 8
    #define TYPE_NIL '0'
    #define TYPE_TRUE 'T'
    #define TYPE_FALSE 'F'
    #define TYPE_FIXNUM 'i'
    #define TYPE_BIGNUM 'l'
    #define TYPE_FLOAT 'f'
    #define TYPE_SYMBOL ':'
    #define TYPE_CLASS 'c'
    #define TYPE_MODULE 'm'
    #define TYPE_LINK '@'
    #define TYPE_SYMLINK ';'
    #define TYPE_STRING '"'
    #define TYPE_REGEXP '/'
    #define TYPE_ARRAY '['
    #define TYPE_HASH '{'
    #define TYPE_HASH_DEF '}'
    #define TYPE_STRUCT 'S'
    #define TYPE_OBJECT 'o'
    #define TYPE_IVAR 'I'
    #define TYPE_UCLASS 'C'
    #define TYPE_EXTENDED 'e'
    #define TYPE_USERDEF 'u'
    #define TYPE_USRMARSHAL 'U'
    

    View Slide

38. marshal.c
    #define MARSHAL_MAJOR 4
    #define MARSHAL_MINOR 8
    #define TYPE_NIL '0'
    #define TYPE_TRUE 'T'
    #define TYPE_FALSE 'F'
    #define TYPE_FIXNUM 'i'
    #define TYPE_BIGNUM 'l'
    #define TYPE_FLOAT 'f'
    #define TYPE_SYMBOL ':'
    #define TYPE_CLASS 'c'
    #define TYPE_MODULE 'm'
    #define TYPE_LINK '@'
    #define TYPE_SYMLINK ';'
    #define TYPE_STRING '"'
    #define TYPE_REGEXP '/'
    #define TYPE_ARRAY '['
    #define TYPE_HASH '{'
    #define TYPE_HASH_DEF '}'
    #define TYPE_STRUCT 'S'
    #define TYPE_OBJECT 'o'
    #define TYPE_IVAR 'I'
    #define TYPE_UCLASS 'C'
    #define TYPE_EXTENDED 'e'
    #define TYPE_USERDEF 'u'
    #define TYPE_USRMARSHAL 'U'
    atomic
    

    View Slide

39. marshal.c
    #define MARSHAL_MAJOR 4
    #define MARSHAL_MINOR 8
    #define TYPE_NIL '0'
    #define TYPE_TRUE 'T'
    #define TYPE_FALSE 'F'
    #define TYPE_FIXNUM 'i'
    #define TYPE_BIGNUM 'l'
    #define TYPE_FLOAT 'f'
    #define TYPE_SYMBOL ':'
    #define TYPE_CLASS 'c'
    #define TYPE_MODULE 'm'
    #define TYPE_LINK '@'
    #define TYPE_SYMLINK ';'
    #define TYPE_STRING '"'
    #define TYPE_REGEXP '/'
    #define TYPE_ARRAY '['
    #define TYPE_HASH '{'
    #define TYPE_HASH_DEF '}'
    #define TYPE_STRUCT 'S'
    #define TYPE_OBJECT 'o'
    #define TYPE_IVAR 'I'
    #define TYPE_UCLASS 'C'
    #define TYPE_EXTENDED 'e'
    #define TYPE_USERDEF 'u'
    #define TYPE_USRMARSHAL 'U'
    atomic
    composite
    

    View Slide

40. marshal.c
    #define MARSHAL_MAJOR 4
    #define MARSHAL_MINOR 8
    #define TYPE_NIL '0'
    #define TYPE_TRUE 'T'
    #define TYPE_FALSE 'F'
    #define TYPE_FIXNUM 'i'
    #define TYPE_BIGNUM 'l'
    #define TYPE_FLOAT 'f'
    #define TYPE_SYMBOL ':'
    #define TYPE_CLASS 'c'
    #define TYPE_MODULE 'm'
    #define TYPE_LINK '@'
    #define TYPE_SYMLINK ';'
    #define TYPE_STRING '"'
    #define TYPE_REGEXP '/'
    #define TYPE_ARRAY '['
    #define TYPE_HASH '{'
    #define TYPE_HASH_DEF '}'
    #define TYPE_STRUCT 'S'
    #define TYPE_OBJECT 'o'
    #define TYPE_IVAR 'I'
    #define TYPE_UCLASS 'C'
    #define TYPE_EXTENDED 'e'
    #define TYPE_USERDEF 'u'
    #define TYPE_USRMARSHAL 'U'
    atomic
    composite
    

    View Slide

41. Objects
    Objects
    

    View Slide

42. marshal.c
    #define MARSHAL_MAJOR 4
    #define MARSHAL_MINOR 8
    #define TYPE_NIL '0'
    #define TYPE_TRUE 'T'
    #define TYPE_FALSE 'F'
    #define TYPE_FIXNUM 'i'
    #define TYPE_BIGNUM 'l'
    #define TYPE_FLOAT 'f'
    #define TYPE_SYMBOL ':'
    #define TYPE_CLASS 'c'
    #define TYPE_MODULE 'm'
    #define TYPE_LINK '@'
    #define TYPE_SYMLINK ';'
    #define TYPE_STRING '"'
    #define TYPE_REGEXP '/'
    #define TYPE_ARRAY '['
    #define TYPE_HASH '{'
    #define TYPE_HASH_DEF '}'
    #define TYPE_STRUCT 'S'
    #define TYPE_OBJECT 'o'
    #define TYPE_IVAR 'I'
    #define TYPE_UCLASS 'C'
    #define TYPE_EXTENDED 'e'
    #define TYPE_USERDEF 'u'
    #define TYPE_USRMARSHAL 'U'
    atomic
    composite
    

    View Slide

43. \x04\bo:\tPost\x1A:\x10@new_recordF:\x10@attributeso:\x1EActiveModel::Attribu
    teSet\x06;\a{\tI\"\aid\x06:\x06ETo:)ActiveModel::Attribute::FromDatabase\n:\
    n@name@\b:\x1C@value_before_type_casti\x06:\n@typeo:EActiveRecord::Connection
    Adapters::SQLite3Adapter::SQLite3Integer\t:\x0F@precision0:\v@scale0:\v@limit
    0:\v@rangeo:\nRange\b:\texclT:\nbeginl-\t\x00\x00\x00\x00\x00\x00\x00\x80:\be
    ndl+\t\x00\x00\x00\x00\x00\x00\x00\x80:\x18@original_attribute0:\v@valuei\x06
    I\"\ntitle\x06;\tTo;\n\n;\v@\x0E;\fI\"\x1CCaching With MessagePack\x06;\tT;\
    ro :\x1EActiveModel::Type::String\n:\n@trueI\"\x06t\x06;\tT:\v@falseI\"\x06f\
    x06 ;\tT;\x0F0;\x100;\x110;\x170;\x18I\"\x1CCaching With MessagePack\x06;\
    tTI\"\x0 Fcreated_at\x06;\tTo;\n\n;\v@\x15;\fU: ...
    

    View Slide

44. 0408 6f3a 0950 6f73 741a 3a10 406e 6577 5f72 6563
    6f72 6446 3a10 4061 7474 7269 6275 7465 736f 3a1e
    4163 7469 7665 4d6f 6465 6c3a 3a41 7474 7269 6275
    7465 5365 7406 3b07 7b09 4922 0769 6406 3a06 4554
    6f3a 2941 6374 6976 654d 6f64 656c 3a3a...
    \x04\bo:\tPost\x1A:\x10@new_recordF:\x10@attributeso:\x1EActiveModel::Attribu
    teSet\x06;\a{\tI\"\aid\x06:\x06ETo:)ActiveModel::Attribute::FromDatabase\n:\
    n@name@\b:\x1C@value_before_type_casti\x06:\n@typeo:EActiveRecord::Connection
    Adapters::SQLite3Adapter::SQLite3Integer\t:\x0F@precision0:\v@scale0:\v@limit
    0:\v@rangeo:\nRange\b:\texclT:\nbeginl-\t\x00\x00\x00\x00\x00\x00\x00\x80:\be
    ndl+\t\x00\x00\x00\x00\x00\x00\x00\x80:\x18@original_attribute0:\v@valuei\x06
    I\"\ntitle\x06;\tTo;\n\n;\v@\x0E;\fI\"\x1CCaching With MessagePack\x06;\tT;\
    ro :\x1EActiveModel::Type::String\n:\n@trueI\"\x06t\x06;\tT:\v@falseI\"\x06f\
    x06 ;\tT;\x0F0;\x100;\x110;\x170;\x18I\"\x1CCaching With MessagePack\x06;\
    tTI\"\x0 Fcreated_at\x06;\tTo;\n\n;\v@\x15;\fU: ...
    

    View Slide

45. 0408 6f3a 0950 6f73 741a 3a10 406e 6577 5f72 6563
    6f72 6446 3a10 4061 7474 7269 6275 7465 736f 3a1e
    4163 7469 7665 4d6f 6465 6c3a 3a41 7474 7269 6275
    7465 5365 7406 3b07 7b09 4922 0769 6406 3a06 4554
    6f3a 2941 6374 6976 654d 6f64 656c 3a3a...
    Version
    

    View Slide

46. 0408 6f3a 0950 6f73 741a 3a10 406e 6577 5f72 6563
    6f72 6446 3a10 4061 7474 7269 6275 7465 736f 3a1e
    4163 7469 7665 4d6f 6465 6c3a 3a41 7474 7269 6275
    7465 5365 7406 3b07 7b09 4922 0769 6406 3a06 4554
    6f3a 2941 6374 6976 654d 6f64 656c 3a3a...
    o :
    length
    o s t
    P
    number of ivars
    

    View Slide

47. 0408 6f3a 0950 6f73 741a 3a10 406e 6577 5f72 6563
    6f72 6446 3a10 4061 7474 7269 6275 7465 736f 3a1e
    4163 7469 7665 4d6f 6465 6c3a 3a41 7474 7269 6275
    7465 5365 7406 3b07 7b09 4922 0769 6406 3a06 4554
    6f3a 2941 6374 6976 654d 6f64 656c 3a3a...
    :
    length
    @new_record
    

    View Slide

48. 0408 6f3a 0950 6f73 741a 3a10 406e 6577 5f72 6563
    6f72 6446 3a10 4061 7474 7269 6275 7465 736f 3a1e
    4163 7469 7665 4d6f 6465 6c3a 3a41 7474 7269 6275
    7465 5365 7406 3b07 7b09 4922 0769 6406 3a06 4554
    6f3a 2941 6374 6976 654d 6f64 656c 3a3a...
    F
    

    View Slide

49. 0408 6f3a 0950 6f73 741a 3a10 406e 6577 5f72 6563
    6f72 6446 3a10 4061 7474 7269 6275 7465 736f 3a1e
    4163 7469 7665 4d6f 6465 6c3a 3a41 7474 7269 6275
    7465 5365 7406 3b07 7b09 4922 0769 6406 3a06 4554
    6f3a 2941 6374 6976 654d 6f64 656c 3a3a...
    

    View Slide

50. variables
    variables
    

    View Slide

51. mystring = "Caching With MessagePack"
    

    View Slide

52. mystring = "Caching With MessagePack"
    mystring.instance_variable_set(:@hidden, true)
    mystring.instance_variable_get(:@hidden)
    #=> true
    

    View Slide

53. mystring = "Caching With MessagePack"
    mystring.instance_variable_set(:@hidden, true)
    mystring.instance_variable_get(:@hidden)
    #=> true
    encoded = Marshal.dump(mystring)
    decoded = Marshal.load(encoded)
    decoded.instance_variable_get(:@hidden)
    #=> ???
    

    View Slide

54. mystring = "Caching With MessagePack"
    mystring.instance_variable_set(:@hidden, true)
    mystring.instance_variable_get(:@hidden)
    #=> true
    encoded = Marshal.dump(mystring)
    decoded = Marshal.load(encoded)
    decoded.instance_variable_get(:@hidden)
    #=> true
    

    View Slide

55. marshal.c
    #define MARSHAL_MAJOR 4
    #define MARSHAL_MINOR 8
    #define TYPE_NIL '0'
    #define TYPE_TRUE 'T'
    #define TYPE_FALSE 'F'
    #define TYPE_FIXNUM 'i'
    #define TYPE_BIGNUM 'l'
    #define TYPE_FLOAT 'f'
    #define TYPE_SYMBOL ':'
    #define TYPE_CLASS 'c'
    #define TYPE_MODULE 'm'
    #define TYPE_LINK '@'
    #define TYPE_SYMLINK ';'
    #define TYPE_STRING '"'
    #define TYPE_REGEXP '/'
    #define TYPE_ARRAY '['
    #define TYPE_HASH '{'
    #define TYPE_HASH_DEF '}'
    #define TYPE_STRUCT 'S'
    #define TYPE_OBJECT 'o'
    #define TYPE_IVAR 'I'
    #define TYPE_UCLASS 'C'
    #define TYPE_EXTENDED 'e'
    #define TYPE_USERDEF 'u'
    #define TYPE_USRMARSHAL 'U'
    atomic
    composite
    

    View Slide

56. links
    links
    

    View Slide

57. myarray = []
    

    View Slide

58. myarray = []
    myarray << myarray
    #=> [[...]]
    

    View Slide

59. myarray = []
    myarray << myarray
    #=> [[...]]
    decoded = Marshal.load(Marshal.dump(myarray))
    #=> ???
    

    View Slide

60. myarray = []
    myarray << myarray
    #=> [[...]]
    decoded = Marshal.load(Marshal.dump(myarray))
    #=> [[...]]
    

    View Slide

61. myarray = []
    myarray << myarray
    #=> [[...]]
    decoded = Marshal.load(Marshal.dump(myarray))
    #=> [[...]]
    decoded.first == decoded
    #=> true
    

    View Slide

62. marshal.c
    #define MARSHAL_MAJOR 4
    #define MARSHAL_MINOR 8
    #define TYPE_NIL '0'
    #define TYPE_TRUE 'T'
    #define TYPE_FALSE 'F'
    #define TYPE_FIXNUM 'i'
    #define TYPE_BIGNUM 'l'
    #define TYPE_FLOAT 'f'
    #define TYPE_SYMBOL ':'
    #define TYPE_CLASS 'c'
    #define TYPE_MODULE 'm'
    #define TYPE_LINK '@'
    #define TYPE_SYMLINK ';'
    #define TYPE_STRING '"'
    #define TYPE_REGEXP '/'
    #define TYPE_ARRAY '['
    #define TYPE_HASH '{'
    #define TYPE_HASH_DEF '}'
    #define TYPE_STRUCT 'S'
    #define TYPE_OBJECT 'o'
    #define TYPE_IVAR 'I'
    #define TYPE_UCLASS 'C'
    #define TYPE_EXTENDED 'e'
    #define TYPE_USERDEF 'u'
    #define TYPE_USRMARSHAL 'U'
    atomic
    composite
    

    View Slide

63. 0408 5b06 4000
    

    View Slide

64. 0408 5b06 4000
    [
    length
    TYPE_ARRAY
    

    View Slide

65. 0408 5b06 4000
    [
    length
    @
    position
    TYPE_LINK
    

    View Slide

66. core type
    core type
    subclasses
    subclasses
    

    View Slide

67. class MyHash < Hash
    end
    

    View Slide

68. class MyHash < Hash
    end
    my_hash = MyHash.new
    #=> {}
    

    View Slide

69. class MyHash < Hash
    end
    my_hash = MyHash.new
    my_hash.class
    #=> MyHash
    

    View Slide

70. class MyHash < Hash
    end
    my_hash = MyHash.new
    encoded = Marshal.dump(my_hash)
    decoded = Marshal.load(encoded)
    #=> {}
    

    View Slide

71. class MyHash < Hash
    end
    my_hash = MyHash.new
    encoded = Marshal.dump(my_hash)
    decoded = Marshal.load(encoded)
    decoded.class
    #=> MyHash
    

    View Slide

72. marshal.c
    #define MARSHAL_MAJOR 4
    #define MARSHAL_MINOR 8
    #define TYPE_NIL '0'
    #define TYPE_TRUE 'T'
    #define TYPE_FALSE 'F'
    #define TYPE_FIXNUM 'i'
    #define TYPE_BIGNUM 'l'
    #define TYPE_FLOAT 'f'
    #define TYPE_SYMBOL ':'
    #define TYPE_CLASS 'c'
    #define TYPE_MODULE 'm'
    #define TYPE_LINK '@'
    #define TYPE_SYMLINK ';'
    #define TYPE_STRING '"'
    #define TYPE_REGEXP '/'
    #define TYPE_ARRAY '['
    #define TYPE_HASH '{'
    #define TYPE_HASH_DEF '}'
    #define TYPE_STRUCT 'S'
    #define TYPE_OBJECT 'o'
    #define TYPE_IVAR 'I'
    #define TYPE_UCLASS 'C'
    #define TYPE_EXTENDED 'e'
    #define TYPE_USERDEF 'u'
    #define TYPE_USRMARSHAL 'U'
    atomic
    composite
    

    View Slide

73. MARSHAL
    MARSHAL
    RUBY DEVELOPER
    RUBY DEVELOPER
    

    View Slide

74. let’s talk about
    msgpack
    msgpack
    

    View Slide

75. before After
    cache
    

    View Slide

76. before After
    

    View Slide

77. We need a format that
    does not encode the universe.
    

    View Slide

78. View Slide

79. encoded = MessagePack.pack({"foo" => "bar"})
    

    View Slide

80. encoded = MessagePack.pack({"foo" => "bar"})
    => "\x81\xA3foo\xA3bar"
    

    View Slide

81. encoded = MessagePack.pack({"foo" => "bar"})
    => "\x81\xA3foo\xA3bar"
    decoded = MessagePack.unpack(encoded)
    => {"foo"=>"bar"}
    

    View Slide

82. Core Types
    

    View Slide

83. Core Types
    atomic
    

    View Slide

84. Core Types
    composite
    

    View Slide

85. Core Types
    No Object Type
    

    View Slide

86. Core Types
    No Object Type
    No ivar Type
    

    View Slide

87. 0408 4922 0866 6f6f 063a 0645 54
    Marshal.dump("foo")
    

    View Slide

88. 0408 4922 0866 6f6f 063a 0645 54
    Marshal.dump("foo")
    foo
    "
    length 3
    

    View Slide

89. 0408 4922 0866 6f6f 063a 0645 54
    Marshal.dump("foo")
    :E
    TYPE_IVAR
    length 1
    T
    “This string’s encoding is UTF-8.”
    

    View Slide

90. a366 6f6f
    MessagePack.pack("foo")
    

    View Slide

91. a366 6f6f
    MessagePack.pack("foo")
    foo
    

    View Slide

92. a366 6f6f
    MessagePack.pack("foo")
    foo
    (String type encoding is UTF-8.)
    

    View Slide

93. a366 6f6f
    MessagePack.pack("foo")
    10100011
    string of length < 31 bytes
    

    View Slide

94. a366 6f6f
    MessagePack.pack("foo")
    10100011
    length 3
    

    View Slide

95. View Slide

96. ActiveRecord::Base
    

    View Slide

97. Object?
    

    View Slide

98. We need a format that
    does not encode the universe.
    whole
    ^
    

    View Slide

99. We need a format that
    does not encode the universe.
    (but let’s us encode parts of it)
    whole
    ^
    

    View Slide

100. factory = MessagePack::Factory.new
     

     View Slide

101. factory = MessagePack::Factory.new
     factory.register_type(
     0x03,
     Date,
     packer: ...,
     unpacker: ...
     )
     Extension Type
     

     View Slide

102. factory = MessagePack::Factory.new
     factory.register_type(
     0x03,
     Date,
     packer: ...,
     unpacker: ...
     )
     type
     

     View Slide

103. factory = MessagePack::Factory.new
     factory.register_type(
     0x03,
     Date,
     packer: ...,
     unpacker: ...
     )
     class
     (matches subclasses!)
     

     View Slide

104. factory = MessagePack::Factory.new
     factory.register_type(
     0x03,
     Date,
     packer: ...,
     unpacker: ...
     )
     serializer/
     deserializer
     

     View Slide

105. packer: ->(date) {
     },
     unpacker: ->(str) {
     }
     

     View Slide

106. packer: ->(date) {
     [date.year,
     date.month,
     date.day].pack("s< C C")
     },
     unpacker: ->(str) {
     }
     Array#pack
     

     View Slide

107. packer: ->(date) {
     [date.year,
     date.month,
     date.day].pack("s< C C")
     },
     unpacker: ->(str) {
     year, month, date =
     str.unpack("s< C C")
     Date.new(year, month, day)
     }
     String#unpack
     

     View Slide

108. packer: ->(date) {
     [date.year,
     date.month,
     date.day].pack("s< C C")
     },
     unpacker: ->(str) {
     year, month, date =
     str.unpack("s< C C")
     Date.new(year, month, day)
     }
     

     View Slide

109. date = Date.new(2022, 9, 9)
     #=> Fri, 09 Sep 2022
     

     View Slide

110. date = Date.new(2022, 9, 9)
     encoded_date = factory.dump(date)
     #=> "\xD6\x03\xE6\a\t\t"
     

     View Slide

111. date = Date.new(2022, 9, 9)
     encoded_date = factory.dump(date)
     #=> "\xD6\x03\xE6\a\t\t"
     d603 e607 0909
     type 3 2022 09 09
     

     View Slide

112. date = Date.new(2022, 9, 9)
     encoded_date = factory.dump(date)
     factory.load(encoded_date)
     #=> Fri, 09 Sep 2022
     

     View Slide

113. date = Date.new(2022, 9, 9)
     encoded_date = factory.dump(date)
     factory.load(encoded_date)
     factory.dump({
     "foo" => [Object.new]
     })
     

     View Slide

114. date = Date.new(2022, 9, 9)
     encoded_date = factory.dump(date)
     factory.load(encoded_date)
     factory.dump({
     "foo" => [Object.new]
     })
     NoMethodError: undefined method
     `to_msgpack' for <#Object:0x...>
     

     View Slide

115. let’s talk about
     what we did
     what we did
     

     View Slide

116. Marshal
     MessagePack
     

     View Slide

117. factory.dump
     

     View Slide

118. MSGPACK
     XX
     factory.dump
     Version Byte Prefix
     

     View Slide

119. MSGPACK
     XX
     rescue NoMethodError
     factory.dump
     

     View Slide

120. MSGPACK
     XX
     rescue NoMethodError
     Marshal.dump
     MARSHAL
     0408
     factory.dump
     

     View Slide

121. ...
     

     View Slide

122. ...
     

     View Slide

123. Symbol
     Time
     DateTime
     Date
     BigDecimal
     

     View Slide

124. Symbol
     Time
     DateTime
     Date
     BigDecimal
     00
     

     View Slide

125. Symbol
     Time
     DateTime
     Date
     BigDecimal
     ActiveRecord::Base
     00 01
     

     View Slide

126. Symbol
     Time
     DateTime
     Date
     BigDecimal
     ActiveRecord::Base
     00 01
     attributes
     cached
     associations
     

     View Slide

127. class Post
     has_many :comments
     end
     class Comment
     belongs_to :post,
     inverse_of: :comments
     end
     

     View Slide

128. [
     [0, [
     [:comments, [[1, [:post, 0]],
     [2, [:post, 0]]]]]],
     [["Post", {"id"=>123, ...}],
     ["Comment", {"post_id"=>123, ..}],
     ["Comment", {"post_id"=>123, ..}]]
     ]
     associations
     records
     

     View Slide

129. c801 2b06 9492 0091 92d7 0063 6f6d 6d65 6e74 7392 9201 9192 d600 706f 7374 0092 0291 92d6 0070 6f73 7400 92a4 506f 7374 84a2 6964 01a5 7469 746c 65b7 4361 6368 696e 6720 5769 7468 6f75 7420 4d61 7273
     6861 6caa 6372 6561 7465 645f 6174 c70f 08f8 cd68 6200 0000 0058 cccd 0d55 5443 aa75 7064 6174 6564 5f61 74c7 0f08 f8cd 6862 0000 0000 58cc cd0d 5554 4392 a743 6f6d 6d65 6e74 84a7 706f 7374 5f69 6401
     a269 6401 aa63 7265 6174 6564 5f61 74c7 0f08 2a79 7862 0000 0000 08b8 4630 5554 43aa 7570 6461 7465 645f 6174 c70f 082a 7978 6200 0000 0008 b846 3055 5443 92a7 436f 6d6d 656e 7484 a269 6402 aa63 7265
     6174 6564 5f61 74c7 0f08 7979 7862 0000 0000 b0c2 7c1f 5554 43aa 7570 6461 7465 645f 6174 c70f 0879 7978 6200 0000 00b0 c27c 1f55 5443 a770 6f73 745f 6964 01
     ActiveRecordCoder + MessagePack
     (303 bytes)
     

     View Slide

130. c801 2b06 9492 0091 92d7 0063 6f6d 6d65 6e74 7392 9201 9192 d600 706f 7374 0092 0291 92d6 0070 6f73 7400 92a4 506f 7374 84a2 6964 01a5 7469 746c 65b7 4361 6368 696e 6720 5769 7468 6f75 7420 4d61 7273
     6861 6caa 6372 6561 7465 645f 6174 c70f 08f8 cd68 6200 0000 0058 cccd 0d55 5443 aa75 7064 6174 6564 5f61 74c7 0f08 f8cd 6862 0000 0000 58cc cd0d 5554 4392 a743 6f6d 6d65 6e74 84a7 706f 7374 5f69 6401
     a269 6401 aa63 7265 6174 6564 5f61 74c7 0f08 2a79 7862 0000 0000 08b8 4630 5554 43aa 7570 6461 7465 645f 6174 c70f 082a 7978 6200 0000 0008 b846 3055 5443 92a7 436f 6d6d 656e 7484 a269 6402 aa63 7265
     6174 6564 5f61 74c7 0f08 7979 7862 0000 0000 b0c2 7c1f 5554 43aa 7570 6461 7465 645f 6174 c70f 0879 7978 6200 0000 00b0 c27c 1f55 5443 a770 6f73 745f 6964 01
     0408 6f3a 0950 6f73 7411 3a10 406e 6577 5f72 6563 6f72 6446 3a10 4061 7474 7269 6275 7465 736f 3a22 4163 7469 7665 4d6f 6465 6c3a 3a4c 617a 7941 7474 7269 6275 7465 5365 740c 3b07 7b09 4922 0769 6406
     3a06 4554 6f3a 2941 6374 6976 654d 6f64 656c 3a3a 4174 7472 6962 7574 653a 3a46 726f 6d44 6174 6162 6173 650a 3a0a 406e 616d 6540 083a 1c40 7661 6c75 655f 6265 666f 7265 5f74 7970 655f 6361 7374 6906
     3a0a 4074 7970 656f 3a45 4163 7469 7665 5265 636f 7264 3a3a 436f 6e6e 6563 7469 6f6e 4164 6170 7465 7273 3a3a 5351 4c69 7465 3341 6461 7074 6572 3a3a 5351 4c69 7465 3349 6e74 6567 6572 093a 0f40 7072
     6563 6973 696f 6e30 3a0b 4073 6361 6c65 303a 0b40 6c69 6d69 7430 3a0b 4072 616e 6765 6f3a 0a52 616e 6765 083a 0965 7863 6c54 3a0a 6265 6769 6e6c 2d09 0000 0000 0000 0080 3a08 656e 646c 2b09 0000 0000
     0000 0080 3a18 406f 7269 6769 6e61 6c5f 6174 7472 6962 7574 6530 3a0b 4076 616c 7565 6906 4922 0a74 6974 6c65 063b 0954 6f3b 0a0a 3b0b 400e 3b0c 4922 1c43 6163 6869 6e67 2057 6974 686f 7574 204d 6172
     7368 616c 063b 0954 3b0d 6f3a 1e41 6374 6976 654d 6f64 656c 3a3a 5479 7065 3a3a 5374 7269 6e67 0a3a 0a40 7472 7565 4922 0674 063b 0954 3a0b 4066 616c 7365 4922 0666 063b 0954 3b0f 303b 1030 3b11 303b
     1730 3b18 4922 1c43 6163 6869 6e67 2057 6974 686f 7574 204d 6172 7368 616c 063b 0954 4922 0f63 7265 6174 6564 5f61 7406 3b09 546f 3b0a 0a3b 0b40 153b 0c49 221f 3230 3232 2d30 342d 3237 2030 353a 3030
     3a34 302e 3233 3135 3931 063b 0954 3b0d 553a 4a41 6374 6976 6552 6563 6f72 643a 3a41 7474 7269 6275 7465 4d65 7468 6f64 733a 3a54 696d 655a 6f6e 6543 6f6e 7665 7273 696f 6e3a 3a54 696d 655a 6f6e 6543
     6f6e 7665 7274 6572 5b09 3a0b 5f5f 7632 5f5f 5b00 5b00 6f3a 2141 6374 6976 6552 6563 6f72 643a 3a54 7970 653a 3a44 6174 6554 696d 6508 3b0f 690b 3b10 303b 1130 3b17 303b 1855 3a20 4163 7469 7665 5375
     7070 6f72 743a 3a54 696d 6557 6974 685a 6f6e 655b 0849 753a 0954 696d 650d 658f 1ec0 a788 8302 063a 097a 6f6e 6549 2208 5554 4306 3b09 4649 2208 5554 4306 3b09 5449 753b 200d 658f 1ec0 a788 8302 063b
     2140 1f49 220f 7570 6461 7465 645f 6174 063b 0954 6f3b 0a0a 3b0b 4023 3b0c 4922 1f32 3032 322d 3034 2d32 3720 3035 3a30 303a 3430 2e32 3331 3539 3106 3b09 543b 0d55 3b1c 5b09 3b1d 5b00 5b00 401c 3b17
     303b 1855 3b1f 5b08 4975 3b20 0d65 8f1e c0a7 8883 0206 3b21 401f 4021 4975 3b20 0d65 8f1e c0a7 8883 0206 3b21 401f 3a0c 4076 616c 7565 737b 0940 0869 0640 0e40 1040 1540 1740 2340 253a 0b40 7479 7065
     737d 0940 0840 0a40 0e40 1140 1540 1840 2340 266f 3a1d 4163 7469 7665 4d6f 6465 6c3a 3a54 7970 653a 3a56 616c 7565 083b 0f30 3b10 303b 1130 3a16 4061 6464 6974 696f 6e61 6c5f 7479 7065 737b 003a 1840
     6465 6661 756c 745f 6174 7472 6962 7574 6573 7b06 4008 6f3b 0a09 3b0b 4008 3b0c 303b 0d40 0a3b 1730 3a13 4063 6173 7465 645f 7661 6c75 6573 7b00 3a12 406d 6174 6572 6961 6c69 7a65 6454 3a17 4061 7373
     6f63 6961 7469 6f6e 5f63 6163 6865 7b06 3a0d 636f 6d6d 656e 7473 553a 3341 6374 6976 6552 6563 6f72 643a 3a41 7373 6f63 6961 7469 6f6e 733a 3a48 6173 4d61 6e79 4173 736f 6369 6174 696f 6e5b 073b 2a5b
     0f5b 073a 0b40 6f77 6e65 7240 005b 073a 1340 6469 7361 626c 655f 6a6f 696e 7346 5b07 3a0c 406c 6f61 6465 6454 5b07 3a0c 4074 6172 6765 745b 076f 3a0c 436f 6d6d 656e 7411 3b06 463b 076f 3b08 0c3b 077b
     0949 220c 706f 7374 5f69 6406 3b09 466f 3b0a 0a3b 0b40 413b 0c69 063b 0d40 0a3b 1730 3b18 6906 4008 6f3b 0a0a 3b0b 4008 3b0c 6906 3b0d 400a 3b17 303b 1869 0640 156f 3b0a 0a3b 0b40 153b 0c49 221f 3230
     3232 2d30 352d 3039 2030 323a 3135 3a30 362e 3830 3939 3431 063b 0954 3b0d 553b 1c5b 093b 1d5b 005b 0040 1c3b 1730 3b18 553b 1f5b 0849 753b 200d 2291 1ec0 d55b 6c3c 063b 2140 1f40 2149 753b 200d 2291
     1ec0 d55b 6c3c 063b 2140 1f40 236f 3b0a 0a3b 0b40 233b 0c49 221f 3230 3232 2d30 352d 3039 2030 323a 3135 3a30 362e 3830 3939 3431 063b 0954 3b0d 553b 1c5b 093b 1d5b 005b 0040 1c3b 1730 3b18 553b 1f5b
     0849 753b 200d 2291 1ec0 d55b 6c3c 063b 2140 1f40 2149 753b 200d 2291 1ec0 d55b 6c3c 063b 2140 1f3b 227b 0940 0869 0640 1540 4540 2340 4f49 220c 706f 7374 5f69 6406 3b09 5469 063b 237d 0940 0840 0a40
     1540 4640 2340 5040 5940 0a40 303b 257b 003b 267b 0640 086f 3b0a 093b 0b40 083b 0c30 3b0d 400a 3b17 303b 277b 003b 2854 3b29 7b06 3a09 706f 7374 553a 3541 6374 6976 6552 6563 6f72 643a 3a41 7373 6f63
     6961 7469 6f6e 733a 3a42 656c 6f6e 6773 546f 4173 736f 6369 6174 696f 6e5b 073b 315b 0c5b 073b 2c40 3e5b 073b 2d46 5b07 3b2e 545b 073b 2f40 005b 073a 1140 7374 616c 655f 7374 6174 6549 2206 3106 3b09
     465b 073a 0d40 7570 6461 7465 6446 5b07 3a17 4061 7373 6f63 6961 7469 6f6e 5f73 636f 7065 303a 0e40 7265 6164 6f6e 6c79 463a 1b40 7072 6576 696f 7573 6c79 5f6e 6577 5f72 6563 6f72 6446 3a0f 4064 6573
     7472 6f79 6564 463a 1c40 6d61 726b 6564 5f66 6f72 5f64 6573 7472 7563 7469 6f6e 463a 1e40 6465 7374 726f 7965 645f 6279 5f61 7373 6f63 6961 7469 6f6e 303a 1e40 5f73 7461 7274 5f74 7261 6e73 6163 7469
     6f6e 5f73 7461 7465 303a 1140 7072 696d 6172 795f 6b65 7940 083a 1440 7374 7269 6374 5f6c 6f61 6469 6e67 463a 1940 7374 7269 6374 5f6c 6f61 6469 6e67 5f6d 6f64 653a 0861 6c6c 6f3b 301b 3b06 463b 076f
     3a1e 4163 7469 7665 4d6f 6465 6c3a 3a41 7474 7269 6275 7465 5365 7406 3b07 7b09 4008 6f3b 0a0a 3b0b 4008 3b0c 6907 3b0d 400a 3b17 303b 1869 0740 156f 3b0a 0a3b 0b40 153b 0c55 3b1f 5b08 4975 3b20 0d22
     911e c08e 0f98 4106 3b21 401f 4021 4975 3b20 0d22 911e c08e 0f98 4106 3b21 401f 3b0d 4046 3b17 303b 1840 7040 236f 3b0a 0a3b 0b40 233b 0c55 3b1f 5b08 4072 4021 4975 3b20 0d22 911e c08e 0f98 4106 3b21
     401f 3b0d 4050 3b17 303b 1840 7540 596f 3b0a 0a3b 0b40 593b 0c69 063b 0d40 0a3b 1730 3b18 6906 3b29 7b06 3b31 553b 325b 073b 315b 0c5b 073b 2c40 6b5b 073b 2d46 5b07 3b2e 545b 073b 2f40 005b 073b 3349
     2206 3106 3b09 465b 073b 3446 5b07 3b35 303b 3646 3b37 543b 3846 3b39 463b 3a30 3b3b 303b 3c40 083b 3d46 3b3e 3b3f 3a1d 406d 7574 6174 696f 6e73 5f66 726f 6d5f 6461 7461 6261 7365 303a 2440 5f6e 6577
     5f72 6563 6f72 645f 6265 666f 7265 5f6c 6173 745f 636f 6d6d 6974 543a 1840 7661 6c69 6461 7469 6f6e 5f63 6f6e 7465 7874 303a 0c40 6572 726f 7273 6f3a 1841 6374 6976 654d 6f64 656c 3a3a 4572 726f 7273
     073a 0a40 6261 7365 406b 3b44 5b00 3a13 405f 746f 7563 685f 7265 636f 7264 543a 1540 5f61 6c72 6561 6479 5f63 616c 6c65 647b 063a 2961 7574 6f73 6176 655f 6173 736f 6369 6174 6564 5f72 6563 6f72 6473
     5f66 6f72 5f70 6f73 7446 3a20 406d 7574 6174 696f 6e73 5f62 6566 6f72 655f 6c61 7374 5f73 6176 656f 3a2a 4163 7469 7665 4d6f 6465 6c3a 3a41 7474 7269 6275 7465 4d75 7461 7469 6f6e 5472 6163 6b65 7207
     3b07 6f3b 4006 3b07 7b09 4008 6f3a 2541 6374 6976 654d 6f64 656c 3a3a 4174 7472 6962 7574 653a 3a46 726f 6d55 7365 720a 3b0b 4008 3b0c 6907 3b0d 400a 3b17 6f3b 0a0a 3b0b 4008 3b0c 303b 0d40 0a3b 1730
     3b18 303b 1869 0740 156f 3b4c 0a3b 0b40 153b 0c40 723b 0d40 463b 176f 3b0a 0a3b 0b40 153b 0c30 3b0d 4046 3b17 303b 1830 3b18 4070 4023 6f3b 4c0a 3b0b 4023 3b0c 4072 3b0d 4050 3b17 6f3b 0a0a 3b0b 4023
     3b0c 303b 0d40 503b 1730 3b18 303b 1840 7540 596f 3b4c 0a3b 0b40 593b 0c69 063b 0d40 0a3b 176f 3b0a 093b 0b40 593b 0c30 3b0d 400a 3b17 303b 1869 063a 1440 666f 7263 6564 5f63 6861 6e67 6573 7b00 3a1f
     405f 636f 6d6d 6974 7465 645f 616c 7265 6164 795f 6361 6c6c 6564 463a 1f40 5f74 7269 6767 6572 5f64 6573 7472 6f79 5f63 616c 6c62 6163 6b46 3a1e 405f 7472 6967 6765 725f 7570 6461 7465 5f63 616c 6c62
     6163 6b46 5b07 3b33 305b 073a 1f40 7265 706c 6163 6564 5f6f 725f 6164 6465 645f 7461 7267 6574 736f 3a08 5365 7406 3a0a 4068 6173 687d 0046 5b07 3a15 4061 7373 6f63 6961 7469 6f6e 5f69 6473 305b 073b
     356f 3a23 436f 6d6d 656e 743a 3a41 6374 6976 6552 6563 6f72 645f 5265 6c61 7469 6f6e 0d3a 0b40 6b6c 6173 7363 0c43 6f6d 6d65 6e74 3a0b 4074 6162 6c65 6f3a 1041 7265 6c3a 3a54 6162 6c65 093b 0b49 220d
     636f 6d6d 656e 7473 063b 0954 3b56 4001 963a 1140 7479 7065 5f63 6173 7465 726f 3a22 4163 7469 7665 5265 636f 7264 3a3a 5479 7065 4361 7374 6572 3a3a 4d61 7006 3b56 4001 963a 1140 7461 626c 655f 616c
     6961 7330 3b22 7b07 3a0e 6578 7465 6e64 696e 675b 003a 0a77 6865 7265 6f3a 2841 6374 6976 6552 6563 6f72 643a 3a52 656c 6174 696f 6e3a 3a57 6865 7265 436c 6175 7365 063a 1040 7072 6564 6963 6174 6573
     5b06 6f3a 1a41 7265 6c3a 3a4e 6f64 6573 3a3a 4571 7561 6c69 7479 073a 0a40 6c65 6674 533a 2041 7265 6c3a 3a41 7474 7269 6275 7465 733a 3a41 7474 7269 6275 7465 073a 0d72 656c 6174 696f 6e40 0197 3a09
     6e61 6d65 4041 3a0b 4072 6967 6874 6f3a 2b41 6374 6976 6552 6563 6f72 643a 3a52 656c 6174 696f 6e3a 3a51 7565 7279 4174 7472 6962 7574 6509 3b0b 4041 3b0c 6906 3b0d 400a 3b17 303b 2e46 3a17 4070 7265
     6469 6361 7465 5f62 7569 6c64 6572 6f3a 2341 6374 6976 6552 6563 6f72 643a 3a50 7265 6469 6361 7465 4275 696c 6465 7207 3b57 6f3a 2041 6374 6976 6552 6563 6f72 643a 3a54 6162 6c65 4d65 7461 6461 7461
     083b 5640 0196 3a10 4061 7265 6c5f 7461 626c 6540 0197 3a10 4072 6566 6c65 6374 696f 6e30 3a0e 4068 616e 646c 6572 735b 0a5b 0763 0853 6574 6f3a 3141 6374 6976 6552 6563 6f72 643a 3a50 7265 6469 6361
     7465 4275 696c 6465 723a 3a41 7272 6179 4861 6e64 6c65 7206 3b67 4001 a15b 0763 0a41 7272 6179 6f3b 6d06 3b67 4001 a15b 0763 1b41 6374 6976 6552 6563 6f72 643a 3a52 656c 6174 696f 6e6f 3a34 4163 7469
     7665 5265 636f 7264 3a3a 5072 6564 6963 6174 6542 7569 6c64 6572 3a3a 5265 6c61 7469 6f6e 4861 6e64 6c65 7200 5b07 630a 5261 6e67 656f 3a31 4163 7469 7665 5265 636f 7264 3a3a 5072 6564 6963 6174 6542
     7569 6c64 6572 3a3a 5261 6e67 6548 616e 646c 6572 063b 6740 01a1 5b07 6310 4261 7369 634f 626a 6563 746f 3a37 4163 7469 7665 5265 636f 7264 3a3a 5072 6564 6963 6174 6542 7569 6c64 6572 3a3a 4261 7369
     634f 626a 6563 7448 616e 646c 6572 063b 6740 01a1 3a17 4064 656c 6567 6174 655f 746f 5f6b 6c61 7373 463a 1340 6675 7475 7265 5f72 6573 756c 7430 3a0d 4072 6563 6f72 6473 305b 073a 0b40 7072 6f78 796f
     3a37 436f 6d6d 656e 743a 3a41 6374 6976 6552 6563 6f72 645f 4173 736f 6369 6174 696f 6e73 5f43 6f6c 6c65 6374 696f 6e50 726f 7879 113a 1140 6173 736f 6369 6174 696f 6e40 363b 5640 0196 3b57 4001 973b
     227b 003b 2e46 3b67 4001 a13b 7146 3b72 303b 7330 3a0a 4074 616b 6530 3a0d 406f 6666 7365 7473 303a 0b40 7363 6f70 6530 5b07 3a11 405f 7761 735f 6c6f 6164 6564 303b 3646 3b37 463b 3846 3b39 463b 3a30
     3b3b 303b 3c40 083b 3d46 3b3e 3b3f
     Marshal
     (4014 bytes)
     

     View Slide

131. Rails cache memcached fill percent
     

     View Slide

132. View Slide

133. Symbol
     Time
     DateTime
     Date
     BigDecimal
     ActiveRecord::Base
     00 01 02
     HashWithIndifferentAccess
     

     View Slide

134. Symbol
     Time
     DateTime
     Date
     BigDecimal
     ActiveRecord::Base
     00 01 02
     HashWithIndifferentAccess < Hash
     core type subclass
     

     View Slide

135. hwia = {}.with_indifferent_access
     

     View Slide

136. hwia = {}.with_indifferent_access
     hwia.class
     #=> HashWithIndifferentAccess
     

     View Slide

137. hwia = {}.with_indifferent_access
     hwia.class
     #=> HashWithIndifferentAccess
     encoded = MessagePack.pack(hwia)
     decoded = MessagePack.unpack(encoded)
     

     View Slide

138. hwia = {}.with_indifferent_access
     hwia.class
     #=> HashWithIndifferentAccess
     encoded = MessagePack.pack(hwia)
     decoded = MessagePack.unpack(encoded)
     decoded.class
     #=> Hash
     

     View Slide

139. /* ext/msgpack/packer.c */
     void msgpack_packer_write_value(msgpack_packer_t* pk, VALUE v)
     {
     switch(rb_type(v)) {
     case T_NIL:
     msgpack_packer_write_nil(pk);
     break;
     case T_HASH:
     msgpack_packer_write_hash_value(pk, v);
     break;
     default:
     msgpack_packer_write_other_value(pk, v);
     }
     }
     ... ...
     matches subclasses!
     v1.4.2
     

     View Slide

140. factory = MessagePack::Factory.new
     

     View Slide

141. factory = MessagePack::Factory.new
     factory.register_type(
     0x7,
     HashWithIndifferentAccess,
     packer: -> (value) {
     },
     unpacker: -> (str) {
     }
     )
     

     View Slide

142. factory = MessagePack::Factory.new
     factory.register_type(
     0x7,
     HashWithIndifferentAccess,
     packer: -> (value) {
     factory.dump(value.to_h)
     },
     unpacker: -> (str) {
     }
     )
     

     View Slide

143. factory = MessagePack::Factory.new
     factory.register_type(
     0x7,
     HashWithIndifferentAccess,
     packer: -> (value) {
     factory.dump(value.to_h)
     },
     unpacker: -> (str) {
     factory.load(value).
     with_indifferent_access
     }
     )
     

     View Slide

144. /* ext/msgpack/packer.c */
     void msgpack_packer_write_value(msgpack_packer_t* pk, VALUE v)
     {
     switch(rb_type(v)) {
     case T_NIL:
     msgpack_packer_write_nil(pk);
     break;
     case T_HASH:
     msgpack_packer_write_hash_value(pk, v);
     break;
     default:
     msgpack_packer_write_other_value(pk, v);
     }
     }
     ... ...
     extension type
     v1.4.2
     

     View Slide

145. /* ext/msgpack/packer.c */
     void msgpack_packer_write_value(msgpack_packer_t* pk, VALUE v)
     {
     switch(rb_type(v)) {
     case T_NIL:
     msgpack_packer_write_nil(pk);
     break;
     case T_HASH:
     msgpack_packer_write_hash_value(pk, v);
     break;
     default:
     msgpack_packer_write_other_value(pk, v);
     }
     }
     ... ...
     this matches first
     v1.4.2
     

     View Slide

146. msgpack 1.4.3 (Jan, 2022)
     

     View Slide

147. Symbol
     Time
     DateTime
     Date
     BigDecimal
     ActiveRecord::Base
     00 01 02 03
     HashWithIndifferentAccess
     ActiveSupport::TimeWithZone
     04 ...
     

     View Slide

148. View Slide

149. Symbol
     Time
     DateTime
     Date
     BigDecimal
     ActiveRecord::Base
     00 01 02 03 7f
     Object
     HashWithIndifferentAccess
     ActiveSupport::TimeWithZone
     04 ...
     

     View Slide

150. class Task
     def initialize(title, content)
     @title, @content = title, content
     end
     def as_pack
     [@title, @content]
     end
     def self.from_pack(payload)
     title, content = payload
     new(title, content)
     end
     end
     “packable”
     

     View Slide

151. MyStruct = Struct.new(:foo, :bar)
     MyStruct.include(Paquito::Struct)
     

     View Slide

152. MyStruct = Struct.new(:foo, :bar)
     MyStruct.include(Paquito::Struct)
     my_struct = MyStruct.new("foo", "bar")
     my_struct.as_pack
     #=> [26450, "foo", "bar"]
     digest
     

     View Slide

153. MyStruct = Struct.new(:foo, :bar)
     MyStruct.include(Paquito::Struct)
     my_struct = MyStruct.new("foo", "bar")
     my_struct.as_pack
     #=> [26450, "foo", "bar"]
     MyStruct.from_pack([26450, "foo", "bar"])
     #=> #
     

     View Slide

154. MyStruct = Struct.new(:foo, :bar)
     MyStruct.include(Paquito::Struct)
     my_struct = MyStruct.new("foo", "bar")
     my_struct.as_pack
     #=> [26450, "foo", "bar"]
     MyStruct.from_pack([26450, "foo", "bar"])
     #=> #
     MyStruct.from_pack([123, "foo", "bar"])
     # MyStruct digests do not match!
     

     View Slide

155. View Slide

156. MessagePack
     

     View Slide

157. Shopify/paquito
     

     View Slide

158. Shopify/paquito
     Jean Boussier
     (@byroot)
     

     View Slide

159. Chris Salzberg
     @shioyama
     Thanks!
     

     View Slide