Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mark Compact GC in MRI

Mark Compact GC in MRI

My presentation about Mark Compact GC in MRI for GitHub Constellation Tokyo

Aaron Patterson

June 13, 2017
Tweet

More Decks by Aaron Patterson

Other Decks in Technology

Transcript

  1. How people build software
    !
    "
    Mark Compact GC in MRI

    View Slide

  2. How people build software
    !
    ͜Μʹͪ͸ʂʂʂ
    2

    View Slide

  3. How people build software
    !
    ࣗݾ঺հ
    3

    View Slide

  4. How people build software
    !
    ΞʔϩϯͰ͢ɻ!
    4

    View Slide

  5. How people build software
    !
    ˏtenderlove
    5

    View Slide

  6. How people build software
    !
    !
    ˏtenderlove
    6

    View Slide

  7. How people build software
    !
    ఱက͕େ޷͖Ͱɺ
    7
    ఱကϥϒΛݺΜͰ΋͍͍Ͱ͢

    View Slide

  8. How people build software
    !
    Ruby Core Team
    Rails Core Team
    8

    View Slide

  9. How people build software
    !
    ೔ຊޠೳྗࢼݧ
    9

    View Slide

  10. How people build software
    !
    ೣɺೋඖ
    10

    View Slide

  11. How people build software
    ! 11
    SEATAC Airport Facebook YouTube

    View Slide

  12. How people build software
    ! 12

    View Slide

  13. How people build software
    ! 13
    Gorbachev Puff Puff Thunderhorse
    ΰϒͪΌΜ

    View Slide

  14. How people build software
    !
    GitHub ͷࣾһ
    14

    View Slide

  15. How people build software
    !
    Ruby & Rails Developer
    15

    View Slide

  16. How people build software
    !
    GitHub & OSS
    16

    View Slide

  17. How people build software
    !
    !
    GitHub Core Values
    17
    • Collaboration
    • Empathy
    • Quality
    • Positive Impact
    • Shipping

    View Slide

  18. How people build software
    !
    Collaboration
    18
    !

    View Slide

  19. How people build software
    !
    Empathy
    19
    !

    View Slide

  20. How people build software
    !
    Quality
    20
    !

    View Slide

  21. How people build software
    !
    Positive Impact
    21
    !

    View Slide

  22. How people build software
    !
    Shipping
    22
    !

    View Slide

  23. How people build software
    !
    GitHub ❤ OSS
    23
    !

    View Slide

  24. How people build software
    !
    GitHub & OSS
    24
    !

    View Slide

  25. How people build software
    !
    Mark / Compact GC
    25

    View Slide

  26. How people build software
    !
    Mark / Compact GC
    26

    View Slide

  27. How people build software
    !
    !
    Roadmap
    27
    Ξϩέʔγϣϯ
    ίϨΫγϣϯ
    ίϯύΫτ

    View Slide

  28. How people build software
    !
    Ξϩέʔγϣϯ
    28

    View Slide

  29. How people build software
    !
    !
    Ξϩέʔγϣϯ
    29
    def hoge
    loop do
    Object.new
    end
    end
    hoge

    View Slide

  30. How people build software
    !
    Ξϩέʔγϣϯ΋
    GCͷ੹೚
    30

    View Slide

  31. How people build software
    !
    `rb_newobj` in gc.c
    31

    View Slide

  32. How people build software
    !
    !
    ϝϞϦྖҬͷϨΠΞ΢τ
    32
    ώʔϓྖҬ
    ϖʔδ ϖʔδ ϖʔδ ϖʔδ ϖʔδ ϖʔδ ϖʔδ ϖʔδ
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj Obj

    View Slide

  33. How people build software
    !
    Free Space = "Fast Path"
    33

    View Slide

  34. How people build software
    !
    New Page = "Slow Path"
    34

    View Slide

  35. How people build software
    !
    !
    ϖʔδͷϨΠΞ΢τ
    35
    ϖʔδ
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    SLOTS
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj

    View Slide

  36. How people build software
    !
    !
    ϖʔδͷϨΠΞ΢τ
    36
    ϖʔδ
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    SLOTS
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj

    View Slide

  37. How people build software
    !
    ཁ໿
    37

    View Slide

  38. How people build software
    !
    heap has_many :pages
    38

    View Slide

  39. How people build software
    !
    page has_many :slots
    39

    View Slide

  40. How people build software
    !
    slot.is_a?(Empty) ||
    slot.is_a?(Object)
    40

    View Slide

  41. How people build software
    !
    ίϨΫγϣϯ
    41

    View Slide

  42. How people build software
    !
    ϚʔΫεΠʔϓ GC
    Mark Sweep GC
    42
    !

    View Slide

  43. How people build software
    !
    ϚʔΫεΠʔϓͷ
    ΞϧΰϦζϜ
    43

    View Slide

  44. How people build software
    !
    !
    Tree of Objects
    44
    Root
    A
    B
    C D
    a = [
    { c: 'd' }
    ]
    Ruby
    Array
    Hash
    String
    Symbol

    View Slide

  45. How people build software
    !
    !
    Tree of Objects
    45
    Root
    A
    B
    C D
    a = [
    { c: 'd' }
    ]
    a = nil
    Ruby

    View Slide

  46. How people build software
    !
    ϚʔΫɹεΠʔϓ
    46
    ͔ͩΒʮϚʔΫεΠʔϓʯɹ
    tenderlove

    View Slide

  47. How people build software
    !
    !
    Mark Phase
    47
    Root
    D
    A
    B
    C
    F
    E
    H
    G

    View Slide

  48. How people build software
    !
    !
    Mark Phase
    48
    Root
    D
    A
    B
    C
    F
    E
    H
    G

    View Slide

  49. How people build software
    !
    !
    Mark Phase
    49
    Root
    D
    A
    B
    C
    F
    E
    H
    G

    View Slide

  50. How people build software
    !
    !
    Mark Phase
    50
    Root
    D
    A
    B
    C
    F
    E
    H
    G

    View Slide

  51. How people build software
    !
    Incremental / Lazy
    51

    View Slide

  52. How people build software
    !
    ίϯύΫγϣϯͷؔ܎
    52

    View Slide

  53. How people build software
    !
    !
    Free Space
    53
    ϖʔδ
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    ϖʔδ
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    ϖʔδ
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj

    View Slide

  54. How people build software
    !
    !
    Free Space
    54
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj

    View Slide

  55. How people build software
    !
    !
    Free Space
    55
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj

    View Slide

  56. How people build software
    !
    ΦϒδΣΫτ͸Ҡಈग़དྷͳ͍
    56

    View Slide

  57. How people build software
    !
    ίϯύΫγϣϯͱ͸
    57

    View Slide

  58. How people build software
    !
    github.com/github/ruby/tree/gc-compact
    58

    View Slide

  59. How people build software
    !
    ίϯύΫγϣϯͷ
    ΞϧΰϦζϜ
    59

    View Slide

  60. How people build software
    !
    "2 Finger Compaction"
    60
    ☝ ☝

    View Slide

  61. How people build software
    !
    !
    Compaction Steps
    61
    ΦϒδΣΫτͷҠಈ
    ϦϑΝϨϯεͷߋ৽

    View Slide

  62. How people build software
    !
    !
    ΦϒδΣΫτͷҠಈ
    62
    1 2 3 4 5 6 7 8 9 a b
    Free Free Free Obj Free Obj Obj Obj Free Free Obj
    ☝ ☝
    Free Pointer Scan Pointer
    1
    2
    3
    5
    Done!
    Address
    Heap

    View Slide

  63. How people build software
    !
    !
    ΦϒδΣΫτҠಈͷલ
    63
    1 2 3 4 5 6 7 8 9 a b
    Free Free Free Obj Free Obj Obj Obj Free Free Obj
    Address
    Heap
    a = { c: 'd' }
    Ruby
    {} :c 'd'

    View Slide

  64. How people build software
    !
    !
    ΦϒδΣΫτҠಈͷޙ
    64
    1 2 3 4 5 6 7 8 9 a b
    Obj Free
    Obj
    Obj
    Obj Free Free
    Obj
    Address
    Heap
    a = { c: 'd' }
    Ruby
    {} :c
    'd'
    1
    2
    3
    5

    View Slide

  65. How people build software
    !
    !
    ϦϑΝϨϯεͷߋ৽
    65
    1 2 3 4 5 6 7 8 9 a b
    Obj Obj
    Obj
    Obj Free Free
    Obj
    Address
    Heap
    a = { c: 'd' }
    Ruby
    {} :c
    'd'
    1
    2
    3
    5

    Done!

    View Slide

  66. How people build software
    !
    !
    ϦϑΝϨϯεͷߋ৽
    66
    1 2 3 4 5 6 7 8 9 a b
    Obj Obj
    Obj
    Obj Free Free
    Obj
    Address
    Heap
    a = { c: 'd' }
    Ruby
    {} :c
    'd'
    1
    2
    3
    5 Free
    Free
    Free
    Free

    View Slide

  67. How people build software
    !
    ίϯύΫτ׬ྃʂʂ
    67

    View Slide

  68. How people build software
    !
    ίϯύΫγϣϯͷ໰୊
    68

    View Slide

  69. How people build software
    !
    !
    ී௨ͷϦϑΝϨϯε
    69
    Hash
    Symbol
    String
    C code

    View Slide

  70. How people build software
    !
    !
    C֦ுͷϦϑΝϨϯε
    70
    Hash
    Symbol
    String
    C code
    rb_gc_mark
    ෆҠಈ
    Ҡಈ
    Ҡ

    View Slide

  71. How people build software
    !
    άϩόϧม਺
    71

    View Slide

  72. How people build software
    !
    !
    file.c in MRI
    72
    separator = rb_obj_freeze(rb_usascii_str_new2("/"));
    /* separates directory parts in path */
    rb_define_const(rb_cFile, "Separator", separator);
    rb_define_const(rb_cFile, "SEPARATOR", separator);
    static VALUE
    rb_file_s_join(VALUE klass, VALUE args)
    {
    return rb_file_join(args, separator);
    }
    άϩόϧม਺

    View Slide

  73. How people build software
    !
    !
    άϩόϧม਺ - SEGV
    73
    $ ruby -e'File::Separator = File::SEPARATOR = nil; GC.start;
    File.join("a", "b")' # BOOM
    $ jruby -e'File::Separator = File::SEPARATOR = nil; GC.start;
    File.join("a", "b")'

    View Slide

  74. How people build software
    !
    General Problem
    74

    View Slide

  75. How people build software
    !
    ݁Ռ
    75

    View Slide

  76. How people build software
    !
    Rails Heap
    76

    View Slide

  77. How people build software
    !
    !
    Compaction Code
    77
    GC.start
    File.open("pre_compact.json", "wb") { |f|
    ObjectSpace.dump_all(output: f)
    }
    GC.start
    p GC.compact
    GC.start
    File.open("post_compact.json", "wb") { |f|
    ObjectSpace.dump_all(output: f)
    }
    $ ruby bin/rails r script.rb

    View Slide

  78. ίϯύΫτͷલ
    F
    U
    P
    Pages
    Number of slots
    0 100 200 300 400

    View Slide

  79. ίϯύΫτͷޙ
    F
    U
    P
    Pages
    Number of slots
    0 100 200 300 400

    View Slide

  80. How people build software
    !
    ίϯύΫτͷલ:
    528 / 552
    80

    View Slide

  81. How people build software
    !
    ίϯύΫτͷޙ:
    167 / 552
    81

    View Slide

  82. How people build software
    !
    Ͱ΋ɺվળͰ͖·͢
    82

    View Slide

  83. How people build software
    !
    GitHub Heap Graphs
    83

    View Slide

  84. View Slide

  85. How people build software
    !
    ·ͱΊ
    85
    !

    View Slide

  86. How people build software
    !
    Why Compact?
    86
    !

    View Slide

  87. How people build software
    !
    OSS Software
    https://github.com/github/ruby/tree/gc-compact
    87
    !

    View Slide

  88. How people build software
    !
    ͋Γ͕ͱ͏͍͟͝·͢ʂ
    88

    View Slide

  89. How people build software
    !
    "

    View Slide