Upgrade to Pro — share decks privately, control downloads, hide ads and more …

My Mom told me that Git doesn't scale

6804f1775cb4babfcc3851298566fbce?s=47 Vicent Martí
November 08, 2012
1.3k

My Mom told me that Git doesn't scale

6804f1775cb4babfcc3851298566fbce?s=128

Vicent Martí

November 08, 2012
Tweet

Transcript

  1. None
  2. Thanks for being here

  3. None
  4. github

  5. github Git hosting: No longer a pain in the ass

  6. github Git hosting: No longer a pain in the ass

    for you. Not for us. Because, goddamnit, if I ever find the guy who invented this thing I’m going to hang him from
  7. Let’s host some Git repos!  file.c  src 

    file.h  README.md  COPYING.md  .git  Bare Repository  HEAD  index  objects  refs git-daemon
  8. OK, now about the web... grit Ruby - Git interface

  9. OK, now about the web... grit  Bare Repo 

    Bare Repo  Bare Repo Ruby - Git interface
  10. 1VM grit storage rails app

  11. nVM storage

  12. nVM storage (GFS)

  13. Rails was making us slow.

  14. None
  15. Literally.

  16. Time to move to Real Hardware

  17. fileservers frontends db

  18. fileservers frontends db ?????????

  19. smoke

  20. None
  21. bert (binary Erlang term)

  22. bert (binary Erlang term) ernie (not an acronym)

  23. chimney (Redis) frontend fileserver smoke grit ernie grit

  24. Horizontal Scaling Vertical Scaling

  25. Horizontal Scaling Vertical Scaling problums wit them gigabits problums wit

    them gigahurtz
  26. them gigabits

  27. bummer x4180 = A LOT.

  28. NetShard       ... alternate network

  29. NetShard       ...  alternate

    network
  30. them gigahurtz

  31. bottleneck: grit

  32. bottleneck: grit solution: git shell out to

  33. bottleneck: git shell out to

  34. bottleneck: git solution: git shell out to shell out to

  35. bottleneck: git solution: git shell out to shell out to

    properly
  36. posix_spawn

  37. posix_spawn Seriously. < 1ms

  38. posix_spawn Seriously. < 1ms The issue is not in “shelling

    out”, the issue is in the spawned process.
  39. None
  40. GUISE

  41. GUISE GUISE

  42. GUISE GUISE GUISE

  43. GUISE GUISE GUISE ...what?

  44. None
  45. Why don’t we take

  46. Why don’t we take the Git binary...

  47. Why don’t we take the Git binary... yeah?

  48. Why don’t we take the Git binary... yeah? and compile

    it as
  49. Why don’t we take the Git binary... yeah? and compile

    it as a library
  50. Why don’t we take the Git binary... yeah? and compile

    it as a library oh... go on...
  51. Why don’t we take the Git binary... yeah? and compile

    it as a library oh... go on... and link that into
  52. Why don’t we take the Git binary... yeah? and compile

    it as a library oh... go on... and link that into our server
  53. Scientific Graph™

  54. Memory Usage Time Scientific Graph™

  55. Memory Usage Time Scientific Graph™

  56. Memory Usage Time Scientific Graph™

  57. Memory Usage Time Scientific Graph™

  58. Memory Usage Time Scientific Graph™

  59. Memory Usage Time Scientific Graph™

  60. None
  61. Well, we didn’t think about

  62. Well, we didn’t think about freeing memory, but...

  63. Well, we didn’t think about freeing memory, but... THIS IS

    THE KIND OF PROBLEM WE COULD SOLVE WITH CGI
  64. Well, we didn’t think about freeing memory, but... THIS IS

    THE KIND OF PROBLEM WE COULD SOLVE WITH CGI IN 1995
  65. Scientific Graph™

  66. Memory Usage Time Scientific Graph™

  67. Memory Usage Time Scientific Graph™

  68. Memory Usage Time Scientific Graph™

  69. Memory Usage Time Scientific Graph™

  70. Memory Usage Time Scientific Graph™

  71. Memory Usage Time Scientific Graph™

  72. Memory Usage Time Scientific Graph™

  73. What do you mean the server died?

  74. die("BUG: non-INDEX attr direction in a bare repo"); What do

    you mean the server died?
  75. die("BUG: non-INDEX attr direction in a bare repo"); die("a bad

    revision is needed"); What do you mean the server died?
  76. die("BUG: non-INDEX attr direction in a bare repo"); die("a bad

    revision is needed"); die("'%s' is not a valid branch name.", name); What do you mean the server died?
  77. die("BUG: non-INDEX attr direction in a bare repo"); die("a bad

    revision is needed"); die("'%s' is not a valid branch name.", name); die("Empty patch. Aborted."); What do you mean the server died?
  78. die("BUG: non-INDEX attr direction in a bare repo"); die("a bad

    revision is needed"); die("'%s' is not a valid branch name.", name); die("Empty patch. Aborted."); die("unable to read index file"); What do you mean the server died?
  79. libgit

  80. libgit2 the “2” means this one frees memory

  81. libgit2 the “2” means this one frees memory NOT ENOUGH

    ABSTRACT FACTORIES
  82. JGit the “J” means this one is in Java ...not

    our thing.
  83. Java a brief timeline New companies don’t use Java because

    it’s not like Unix 1995 New companies use Java because it’s new and shiny 1997 New companies don’t use Java because it’s ooooooold 2005 New companies use the JVM because 2011
  84. Java a brief timeline New companies don’t use Java because

    it’s not like Unix 1995 New companies use Java because it’s new and shiny 1997 New companies don’t use Java because it’s ooooooold 2005 New companies use the JVM because 2011 github
  85. If you think you understand the JVM, you are either:

  86. If you think you understand the JVM, you are either:

    a) Very smart
  87. If you think you understand the JVM, you are either:

    a) Very smart b) Very wrong
  88. If you think you understand the JVM, you are either:

    a) Very smart b) Very wrong
  89. Some people think that github is a Rails shop Ruby

    shop. or even a
  90. Some people think that github is a Rails shop Ruby

    shop. or even a github is a Unix shop and everything else is just a detail.
  91. libgit2 So,

  92. libgit2 a brief timeline Shawn Pearce The Past

  93. libgit2 a brief timeline Shawn Pearce myself The Past

  94. libgit2 a brief timeline Shawn Pearce myself myself (about to

    have a mental breakdown) The Past
  95. libgit2 a brief timeline Shawn Pearce myself myself (about to

    have a mental breakdown) myself (having a mental breakdown) The Past
  96. libgit2 a brief timeline Shawn Pearce myself myself (about to

    have a mental breakdown) myself (having a mental breakdown) myself (reaching Git nirvana) The Past
  97. libgit2 a brief timeline Shawn Pearce myself myself (about to

    have a mental breakdown) myself (having a mental breakdown) myself (reaching Git nirvana) The Past Russell Belfer Carlos Martín Michael Schubert Ben Straub real contributors
  98. libgit2 a brief timeline ?

  99. libgit2 a brief timeline ?

  100. libgit2 a brief timeline ? 1.0 release

  101. libgit2

  102. libgit2

  103. Good Heavens, just look at the time. It’s NoSQL o’clock

    NoSQL NoSQL NoSQL NoSQL NoSQL NoSQL NoSQL NoSQL
  104. None
  105. ...do you even

  106. ...do you even mongo?

  107. Key-Value Stores The Magic of If you wish upon a

    star, and have a pure heart...
  108. Key-Value Stores The Magic of If you wish upon a

    star, and have a pure heart... Anything can be a Key-Value store!
  109. id name state lat 13 San Francisco CA 24 24

    Phoenix AZ 33 7 Denver CO 40 8 Caribou ME 47 2 Los Angeles CA 22 SELECT * FROM CITIES WHERE name = ‘San Francisco’ Key-Value Stores The Magic of
  110. Git is queried like a Key-Value Store But it is

    not a Key-Value store git show f3c896c1949476e85abc0d75bb2143656a9580a6
  111. a b r i e f i n t r

    o d u c t i o n t o t h e G i t d a t a m o d e l
  112. None
  113. None
  114.  file.c  src  file.h  README.md  COPYING.md

  115.  file.c  src  file.h  README.md  COPYING.md

    tree src/ README.md COPYING.md tree file.c file.h blob blob blob blob
  116. commit parent tree T metadata

  117. commit T commit T commit T commit T commit T

    commit T Behold, a graph.
  118. None
  119. Well that was easy.

  120. None
  121. master Oh god kill me

  122. Li le known torture methods:

  123. None
  124. warning: the rabbit hole is pretty deep

  125. None
  126. warning: git totally wasn’t designed for this

  127. Git doesn’t give a #!%$ about CAP

  128. None
  129. Number of hops on a complex query 1,000,000

  130. Number of hops on a complex query 1,000,000 Required hops

    for a successful query 1,000,000
  131. Number of hops on a complex query 1,000,000 Required hops

    for a successful query 1,000,000 Replica count to ensure 100% availability a metric shitton
  132. We could fix it.

  133. We could fix it. But we won’t.

  134. GitRPC

  135. GitRPC Less.

  136. GitRPC Rugged libgit2 server Ruby Ruby C

  137. chimney (Redis) frontend fileserver smoke grit ernie grit GitRPC GitRPC

  138. chimney (Redis) frontend fileserver GitRPC GitRPC server client

  139. New serialization protocol Banana Pack MessagePack + more

  140. New serialization protocol Banana Pack MessagePack + more mochilo

  141. New serialization protocol Banana Pack MessagePack + more mochilo Banana

    Phone
  142. evolutionary (disappointing?)

  143. Summary:

  144. Ruby C Unix Unix Unix Unix Unix Unix Unix Unix

    Boring Boring Boring Boring Boring Summary:
  145. Use the most reliable tools you know.

  146. Challenge yourself to build the simplest thing. Not because it’s

    easy, but because it works.
  147. Innovate where it really ma ers.

  148. revolutionary product create a revolutionary backend. not a

  149. None
  150. Q: Does Git scale?

  151. Q: Does Git scale? A: Who cares?

  152. None