Mark Compact GC in MRI

My presentation about Mark Compact GC in MRI for GitHub Constellation Tokyo

Transcript

1. How people build software
   !
   "
   Mark Compact GC in MRI
   

   View Slide

2. How people build software
   !
   ͜Μʹͪ͸ʂʂʂ
   2
   

   View Slide

3. How people build software
   !
   ࣗݾ঺հ
   3
   

   View Slide

4. How people build software
   !
   ΞʔϩϯͰ͢ɻ!
   4
   

   View Slide

5. How people build software
   !
   ˏtenderlove
   5
   

   View Slide

6. How people build software
   !
   !
   ˏtenderlove
   6
   

   View Slide

7. How people build software
   !
   ఱက͕େ޷͖Ͱɺ
   7
   ఱကϥϒΛݺΜͰ΋͍͍Ͱ͢
   

   View Slide

8. How people build software
   !
   Ruby Core Team
   Rails Core Team
   8
   

   View Slide

9. How people build software
   !
   ೔ຊޠೳྗࢼݧ
   9
   

   View Slide

10. How people build software
    !
    ೣɺೋඖ
    10
    

    View Slide

11. How people build software
    ! 11
    SEATAC Airport Facebook YouTube
    

    View Slide

12. How people build software
    ! 12
    

    View Slide

13. How people build software
    ! 13
    Gorbachev Puff Puff Thunderhorse
    ΰϒͪΌΜ
    

    View Slide

14. How people build software
    !
    GitHub ͷࣾһ
    14
    

    View Slide

15. How people build software
    !
    Ruby & Rails Developer
    15
    

    View Slide

16. How people build software
    !
    GitHub & OSS
    16
    

    View Slide

17. How people build software
    !
    !
    GitHub Core Values
    17
    • Collaboration
    • Empathy
    • Quality
    • Positive Impact
    • Shipping
    

    View Slide

18. How people build software
    !
    Collaboration
    18
    !
    

    View Slide

19. How people build software
    !
    Empathy
    19
    !
    

    View Slide

20. How people build software
    !
    Quality
    20
    !
    

    View Slide

21. How people build software
    !
    Positive Impact
    21
    !
    

    View Slide

22. How people build software
    !
    Shipping
    22
    !
    

    View Slide

23. How people build software
    !
    GitHub ❤ OSS
    23
    !
    

    View Slide

24. How people build software
    !
    GitHub & OSS
    24
    !
    

    View Slide

25. How people build software
    !
    Mark / Compact GC
    25
    

    View Slide

26. How people build software
    !
    Mark / Compact GC
    26
    

    View Slide

27. How people build software
    !
    !
    Roadmap
    27
    Ξϩέʔγϣϯ
    ίϨΫγϣϯ
    ίϯύΫτ
    

    View Slide

28. How people build software
    !
    Ξϩέʔγϣϯ
    28
    

    View Slide

29. How people build software
    !
    !
    Ξϩέʔγϣϯ
    29
    def hoge
    loop do
    Object.new
    end
    end
    hoge
    

    View Slide

30. How people build software
    !
    Ξϩέʔγϣϯ΋
    GCͷ੹೚
    30
    

    View Slide

31. How people build software
    !
    `rb_newobj` in gc.c
    31
    

    View Slide

32. How people build software
    !
    !
    ϝϞϦྖҬͷϨΠΞ΢τ
    32
    ώʔϓྖҬ
    ϖʔδ ϖʔδ ϖʔδ ϖʔδ ϖʔδ ϖʔδ ϖʔδ ϖʔδ
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj Obj
    

    View Slide

33. How people build software
    !
    Free Space = "Fast Path"
    33
    

    View Slide

34. How people build software
    !
    New Page = "Slow Path"
    34
    

    View Slide

35. How people build software
    !
    !
    ϖʔδͷϨΠΞ΢τ
    35
    ϖʔδ
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    SLOTS
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    

    View Slide

36. How people build software
    !
    !
    ϖʔδͷϨΠΞ΢τ
    36
    ϖʔδ
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    SLOTS
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    

    View Slide

37. How people build software
    !
    ཁ໿
    37
    

    View Slide

38. How people build software
    !
    heap has_many :pages
    38
    

    View Slide

39. How people build software
    !
    page has_many :slots
    39
    

    View Slide

40. How people build software
    !
    slot.is_a?(Empty) ||
    slot.is_a?(Object)
    40
    

    View Slide

41. How people build software
    !
    ίϨΫγϣϯ
    41
    

    View Slide

42. How people build software
    !
    ϚʔΫεΠʔϓ GC
    Mark Sweep GC
    42
    !
    

    View Slide

43. How people build software
    !
    ϚʔΫεΠʔϓͷ
    ΞϧΰϦζϜ
    43
    

    View Slide

44. How people build software
    !
    !
    Tree of Objects
    44
    Root
    A
    B
    C D
    a = [
    { c: 'd' }
    ]
    Ruby
    Array
    Hash
    String
    Symbol
    

    View Slide

45. How people build software
    !
    !
    Tree of Objects
    45
    Root
    A
    B
    C D
    a = [
    { c: 'd' }
    ]
    a = nil
    Ruby
    

    View Slide

46. How people build software
    !
    ϚʔΫɹεΠʔϓ
    46
    ͔ͩΒʮϚʔΫεΠʔϓʯɹ
    tenderlove
    

    View Slide

47. How people build software
    !
    !
    Mark Phase
    47
    Root
    D
    A
    B
    C
    F
    E
    H
    G
    

    View Slide

48. How people build software
    !
    !
    Mark Phase
    48
    Root
    D
    A
    B
    C
    F
    E
    H
    G
    

    View Slide

49. How people build software
    !
    !
    Mark Phase
    49
    Root
    D
    A
    B
    C
    F
    E
    H
    G
    

    View Slide

50. How people build software
    !
    !
    Mark Phase
    50
    Root
    D
    A
    B
    C
    F
    E
    H
    G
    

    View Slide

51. How people build software
    !
    Incremental / Lazy
    51
    

    View Slide

52. How people build software
    !
    ίϯύΫγϣϯͷؔ܎
    52
    

    View Slide

53. How people build software
    !
    !
    Free Space
    53
    ϖʔδ
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    ϖʔδ
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    ϖʔδ
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    

    View Slide

54. How people build software
    !
    !
    Free Space
    54
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    

    View Slide

55. How people build software
    !
    !
    Free Space
    55
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    FREE
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    Obj
    

    View Slide

56. How people build software
    !
    ΦϒδΣΫτ͸Ҡಈग़དྷͳ͍
    56
    

    View Slide

57. How people build software
    !
    ίϯύΫγϣϯͱ͸
    57
    

    View Slide

58. How people build software
    !
    github.com/github/ruby/tree/gc-compact
    58
    

    View Slide

59. How people build software
    !
    ίϯύΫγϣϯͷ
    ΞϧΰϦζϜ
    59
    

    View Slide

60. How people build software
    !
    "2 Finger Compaction"
    60
    ☝ ☝
    

    View Slide

61. How people build software
    !
    !
    Compaction Steps
    61
    ΦϒδΣΫτͷҠಈ
    ϦϑΝϨϯεͷߋ৽
    

    View Slide

62. How people build software
    !
    !
    ΦϒδΣΫτͷҠಈ
    62
    1 2 3 4 5 6 7 8 9 a b
    Free Free Free Obj Free Obj Obj Obj Free Free Obj
    ☝ ☝
    Free Pointer Scan Pointer
    1
    2
    3
    5
    Done!
    Address
    Heap
    

    View Slide

63. How people build software
    !
    !
    ΦϒδΣΫτҠಈͷલ
    63
    1 2 3 4 5 6 7 8 9 a b
    Free Free Free Obj Free Obj Obj Obj Free Free Obj
    Address
    Heap
    a = { c: 'd' }
    Ruby
    {} :c 'd'
    

    View Slide

64. How people build software
    !
    !
    ΦϒδΣΫτҠಈͷޙ
    64
    1 2 3 4 5 6 7 8 9 a b
    Obj Free
    Obj
    Obj
    Obj Free Free
    Obj
    Address
    Heap
    a = { c: 'd' }
    Ruby
    {} :c
    'd'
    1
    2
    3
    5
    

    View Slide

65. How people build software
    !
    !
    ϦϑΝϨϯεͷߋ৽
    65
    1 2 3 4 5 6 7 8 9 a b
    Obj Obj
    Obj
    Obj Free Free
    Obj
    Address
    Heap
    a = { c: 'd' }
    Ruby
    {} :c
    'd'
    1
    2
    3
    5
    ☝
    Done!
    

    View Slide

66. How people build software
    !
    !
    ϦϑΝϨϯεͷߋ৽
    66
    1 2 3 4 5 6 7 8 9 a b
    Obj Obj
    Obj
    Obj Free Free
    Obj
    Address
    Heap
    a = { c: 'd' }
    Ruby
    {} :c
    'd'
    1
    2
    3
    5 Free
    Free
    Free
    Free
    

    View Slide

67. How people build software
    !
    ίϯύΫτ׬ྃʂʂ
    67
    

    View Slide

68. How people build software
    !
    ίϯύΫγϣϯͷ໰୊
    68
    

    View Slide

69. How people build software
    !
    !
    ී௨ͷϦϑΝϨϯε
    69
    Hash
    Symbol
    String
    C code
    
    

    View Slide

70. How people build software
    !
    !
    C֦ுͷϦϑΝϨϯε
    70
    Hash
    Symbol
    String
    C code
    rb_gc_mark
    ෆҠಈ
    Ҡಈ
    Ҡ
    ಈ
    

    View Slide

71. How people build software
    !
    άϩόϧม਺
    71
    

    View Slide

72. How people build software
    !
    !
    file.c in MRI
    72
    separator = rb_obj_freeze(rb_usascii_str_new2("/"));
    /* separates directory parts in path */
    rb_define_const(rb_cFile, "Separator", separator);
    rb_define_const(rb_cFile, "SEPARATOR", separator);
    static VALUE
    rb_file_s_join(VALUE klass, VALUE args)
    {
    return rb_file_join(args, separator);
    }
    άϩόϧม਺
    

    View Slide

73. How people build software
    !
    !
    άϩόϧม਺ - SEGV
    73
    $ ruby -e'File::Separator = File::SEPARATOR = nil; GC.start;
    File.join("a", "b")' # BOOM
    $ jruby -e'File::Separator = File::SEPARATOR = nil; GC.start;
    File.join("a", "b")'
    

    View Slide

74. How people build software
    !
    General Problem
    74
    

    View Slide

75. How people build software
    !
    ݁Ռ
    75
    

    View Slide

76. How people build software
    !
    Rails Heap
    76
    

    View Slide

77. How people build software
    !
    !
    Compaction Code
    77
    GC.start
    File.open("pre_compact.json", "wb") { |f|
    ObjectSpace.dump_all(output: f)
    }
    GC.start
    p GC.compact
    GC.start
    File.open("post_compact.json", "wb") { |f|
    ObjectSpace.dump_all(output: f)
    }
    $ ruby bin/rails r script.rb
    

    View Slide

78. ίϯύΫτͷલ
    F
    U
    P
    Pages
    Number of slots
    0 100 200 300 400
    

    View Slide

79. ίϯύΫτͷޙ
    F
    U
    P
    Pages
    Number of slots
    0 100 200 300 400
    

    View Slide

80. How people build software
    !
    ίϯύΫτͷલ:
    528 / 552
    80
    

    View Slide

81. How people build software
    !
    ίϯύΫτͷޙ:
    167 / 552
    81
    

    View Slide

82. How people build software
    !
    Ͱ΋ɺվળͰ͖·͢
    82
    

    View Slide

83. How people build software
    !
    GitHub Heap Graphs
    83
    

    View Slide

84. View Slide

85. How people build software
    !
    ·ͱΊ
    85
    !
    

    View Slide

86. How people build software
    !
    Why Compact?
    86
    !
    

    View Slide

87. How people build software
    !
    OSS Software
    https://github.com/github/ruby/tree/gc-compact
    87
    !
    

    View Slide

88. How people build software
    !
    ͋Γ͕ͱ͏͍͟͝·͢ʂ
    88
    

    View Slide

89. How people build software
    !
    "
    

    View Slide