Scaling Git

Atlassian Summit 2014 - Scaling Git

Transcript

1. #summit14
2. STEFAN SAASEN • DEVELOPMENT MANAGER • ATLASSIAN • @STEFANSAASEN Scaling Git
8. Source View - git cat-file
13. Git is fundamentally a content-addressable filesystem with a VCS user interface written on top of it Pro Git Book, Section: Git Inter nals ” “
14. GIT UNDER THE HOOD $> tree .git/objects .git/objects !"" info #"" pack ! 2 directories
15. GIT UNDER THE HOOD git add some-file.txt
16. GIT UNDER THE HOOD $> tree .git/objects .git/objects !"" e4 $ #"" 3a6ac59164adadac854d591001bbb10086f37d !"" info #"" pack ! 3 directories, 1 file zlib compressed SHA1
17. GIT UNDER THE HOOD git commit -m "First commit"
18. GIT UNDER THE HOOD $> tree .git/objects .git/objects !"" 13 $ #"" 1e360ae1a0c08acd18182c6160af6a83e0d22f !"" 31 $ #"" 995f2d03aa31ee97ee2e814c9f0b0ffd814316 !"" e4 $ #"" 3a6ac59164adadac854d591001bbb10086f37d !"" info #"" pack ! 5 directories, 3 files
19. GIT UNDER THE HOOD $> tree .git/objects .git/objects !"" 13 $ #"" 1e360ae1a0c08acd18182c6160af6a83e0d22f !"" 31 $ #"" 995f2d03aa31ee97ee2e814c9f0b0ffd814316 !"" e4 $ #"" 3a6ac59164adadac854d591001bbb10086f37d !"" info #"" pack ! 5 directories, 3 files Blob
20. GIT UNDER THE HOOD $> tree .git/objects .git/objects !"" 13 $ #"" 1e360ae1a0c08acd18182c6160af6a83e0d22f !"" 31 $ #"" 995f2d03aa31ee97ee2e814c9f0b0ffd814316 !"" e4 $ #"" 3a6ac59164adadac854d591001bbb10086f37d !"" info #"" pack ! 5 directories, 3 files Blob Tree
21. GIT UNDER THE HOOD $> tree .git/objects .git/objects !"" 13 $ #"" 1e360ae1a0c08acd18182c6160af6a83e0d22f !"" 31 $ #"" 995f2d03aa31ee97ee2e814c9f0b0ffd814316 !"" e4 $ #"" 3a6ac59164adadac854d591001bbb10086f37d !"" info #"" pack ! 5 directories, 3 files Blob Tree Commit
22. GIT UNDER THE HOOD echo "// Comment" >> some-file.txt
23. GIT UNDER THE HOOD git add some-file.txt
24. GIT UNDER THE HOOD $> tree .git/objects .git/objects !"" 13 $ #"" 1e360ae1a0c08acd18182c6160af6a83e0d22f !"" 31 $ #"" 995f2d03aa31ee97ee2e814c9f0b0ffd814316 !"" c1 $ #"" 9e6823e34980033917b6427f3e245ce2102e6e !"" e4 $ #"" 3a6ac59164adadac854d591001bbb10086f37d ! 6 directories, 4 files Entirely new BLOB
25. wat?
26. GIT UNDER THE HOOD git gc
27. GIT UNDER THE HOOD $> tree .git/objects .git/objects !"" info $ #"" packs #"" pack !"" pack-7475314b451a882d77b1535d215def8bad0f4306.idx #"" pack-7475314b451a882d77b1535d215def8bad0f4306.pack ! 2 directories, 3 files
28. GIT UNDER THE HOOD Loose Objects
29. GIT UNDER THE HOOD 1.zlib compressed 2.Delta encoded Packfile Loose Objects
30. but...
31. CPU Replace Graph 0 200000 400000 600000 800000 1407989425 1407989450 1407989475 1407989500 1407989525 1407989550 1407989575 user syst CPU
32. Memory 0 100 200 300 400 1407989425 1407989450 1407989475 1407989500 1407989525 1407989550 1407989575 MiB Memory
33. IO Replace Graph 0 5 10 15 20 1407989440 1407989460 1407989480 1407989500 1407989520 1407989 read write Disk I/O
34. What's your point?
36. git clone/fetch
37. generates a packfile git clone/fetch
38. git clone/fetch every time
39. Clone 800 stash.atlassian.com Fetch 1,200
40. This is what we learned so far.
41. 1. SCM Cache
42. CACHE git clone 1
43. git clone 2 CACHE
44. 2. Sizing is important
45. You need sufficient hardware
46. 768MiB Memory budget
47. 5 GiB Memory budget
48. 768MiB 5 GiB Memory budget
49. 3. Limits
50. Limits
51. 4. Continuous Integration
52. What do you have?
53. This is what I've got.
54. Ok, here is what I've got. Give me everything that's new.
55. Here you go!
56. Don't worry. I'm up to date.
57. Caption goes here
58. Avoid Polling.
59. SCM Cache can also cache ref advertisements
60. Consider shallow clones.
61. 5. Update
62. Use recent versions of
64. Stash Data Center
65. RDBMS FS
66. RDBMS NFS
67. Performance at scale
68. RDBMS NFS
69. Scaling Git
70. Scaling Git • Git hosting operations are expensive.
71. Scaling Git • Git hosting operations are expensive. • Properly size the hardware for the workload that you expect.
72. Scaling Git • Git hosting operations are expensive. • Properly size the hardware for the workload that you expect. • Avoid polling when you do a lot of builds, enable caching of ref advertisements when you can't.
73. Scaling Git • Git hosting operations are expensive. • Properly size the hardware for the workload that you expect. • Avoid polling when you do a lot of builds, enable caching of ref advertisements when you can't. • Prefer shallow clones
74. Scaling Git • Git hosting operations are expensive. • Properly size the hardware for the workload that you expect. • Avoid polling when you do a lot of builds, enable caching of ref advertisements when you can't. • Prefer shallow clones • Limits are in place to keep your Stash server running
75. Scaling Git • Git hosting operations are expensive. • Properly size the hardware for the workload that you expect. • Avoid polling when you do a lot of builds, enable caching of ref advertisements when you can't. • Prefer shallow clones • Limits are in place to keep your Stash server running • Stash Data Center allows you to scale out and have high availability
76. Stash Data Center Beta program Sign up!
77. Thank you! STEFAN SAASEN • DEVELOPMENT MANAGER • ATLASSIAN • @STEFANSAASEN