How GitLab scaled Git
access with a Go service
Oswaldo Ferreira (@olsfer)
Backend Engineer at GitLab
Slide 2
Slide 2 text
Hi
• Backend Engineer at GitLab Source Code Team
• Merge Requests
• Code Review
• Web IDE
• Enjoy hunting performance issues at GitLab.com
• Moved to São Paulo about a year ago ✈
Slide 3
Slide 3 text
Today we'll cover
• Introduction: What GitLab is
• How GitLab use Git
• How GitLab scaled Git storage and access
• Limitations
• Gitaly
• gRPC, Protocol Buffers and Prometheus monitoring
• How Gitaly fits into GitLab
Slide 4
Slide 4 text
Introduction (GitLab)
• Single application for the whole DevOps lifecycle
• Git repository hosting
• Code reviewing
• CI/CD
• Monitoring
• Security Testing
• Etc
• Open source projects written in Ruby, Javascript and Go
Slide 5
Slide 5 text
No content
Slide 6
Slide 6 text
Introduction (GitLab)
• Started as a self-hosted application: You host your own
GitLab instance
• GitLab.com (SaaS) now handles about 3k Git operations per
second (with around 9 million projects)
• GitLab.com runs at GitLab Enterprise Edition, which has its
stable packages released (for self-hosted clients) every 22nd
Slide 7
Slide 7 text
A bit of history
• Early days of GitLab.com
• Most of the application in a single server (Unicorn,
Sidekiq, Git storage)
• Easy to deploy and maintain
• Only vertical scaling
• Out of options for continuing scaling GitLab.com vertically
• Horizontal scaling had to be made possible
Slide 8
Slide 8 text
No content
Slide 9
Slide 9 text
No content
Slide 10
Slide 10 text
How applications access
Git
• Libgit2 through Rugged
• Libgit2 is a C implementation of Git core methods
• Rugged is Ruby binding for libgit2 (so everyone could
contribute)
• Directly through Git command-line
• A consolidated internal interface (Ruby) to interact with
Git was built through time (reading and writing)
Ideas & Decisions
• Concentrate all Git access logic within a single codebase
(acting as a "git database")
• gRPC and Protocol Buffers (cross-platform RPC and well
defined API)
• Written in Go
• Prometheus for monitoring
Slide 27
Slide 27 text
git-1.server
unicorn-1.server
gRPC
Make git work "locally"
return the results over the network
• NFS servers become Gitaly servers
• Gitaly has direct disk access (no more NFS latency)
Slide 28
Slide 28 text
Development
• New team to develop Gitaly
• Slowly rollout and use it both on staging and production
before Gitaly 1.0 (using feature flags)
Slide 29
Slide 29 text
No content
Slide 30
Slide 30 text
No content
Slide 31
Slide 31 text
No content
Slide 32
Slide 32 text
• Open source message exchanging framework
• Used by Slack and other beauties
• Low latency, highly scalable
• HTTP/2
• Protocol buffers as a descriptive language for interfaces
Advantages for our
scenario
• Mature interface for interacting with Git
• Server to server binary message streaming
• Numbered fields are powerful for versioning (backward
compatible by default)
• Language interoperability: Ruby client, Go server
Slide 42
Slide 42 text
Right, but can it ?
Slide 43
Slide 43 text
Collecting data
Slide 44
Slide 44 text
• Open source written in
• Time series based monitoring
• Own query language (PromQL)
• Handles alerting
• Great for monitoring (not great for general logging)
• Error ratio, Request ratio, cache hit/miss
Slide 45
Slide 45 text
No content
Slide 46
Slide 46 text
No content
Slide 47
Slide 47 text
No content
Slide 48
Slide 48 text
No content
Slide 49
Slide 49 text
sum()
• Faster message exchanging
• Reliable typed interfaces between client/server
• More visibility
• Self-contained logic
• Higher entry barrier for contributions
• Additional complexity for maintaining
• Funny enough: Higher visibility made us slower at first