Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Making GitLab Faster
Search
Yorick Peterse
June 04, 2016
Programming
2
460
Making GitLab Faster
Talk given at RubyC 2016
Yorick Peterse
June 04, 2016
Tweet
Share
More Decks by Yorick Peterse
See All by Yorick Peterse
Garbage Collection Crash Course
yorickpeterse
1
380
Rubinius & The Eternal Yak
yorickpeterse
1
260
Oga
yorickpeterse
3
200
Parsing for Humans
yorickpeterse
2
99
Other Decks in Programming
See All in Programming
AIプログラマーDevinは PHPerの夢を見るか?
shinyasaita
1
220
Startups on Rails in Past, Present and Future–Irina Nazarova, RailsConf 2025
irinanazarova
0
110
Deep Dive into ~/.claude/projects
hiragram
14
2.5k
Is Xcode slowly dying out in 2025?
uetyo
1
270
「Cursor/Devin全社導入の理想と現実」のその後
saitoryc
0
820
git worktree × Claude Code × MCP ~生成AI時代の並列開発フロー~
hisuzuya
1
570
設計やレビューに悩んでいるPHPerに贈る、クリーンなオブジェクト設計の指針たち
panda_program
6
2.1k
チームのテスト力を総合的に鍛えて品質、スピード、レジリエンスを共立させる/Testing approach that improves quality, speed, and resilience
goyoki
5
880
脱Riverpod?fqueryで考える、TanStack Queryライクなアーキテクチャの可能性
ostk0069
0
140
すべてのコンテキストを、 ユーザー価値に変える
applism118
3
1.3k
チームで開発し事業を加速するための"良い"設計の考え方 @ サポーターズCoLab 2025-07-08
agatan
1
420
Goで作る、開発・CI環境
sin392
0
230
Featured
See All Featured
Intergalactic Javascript Robots from Outer Space
tanoku
271
27k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.5k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
34
3.1k
It's Worth the Effort
3n
185
28k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.4k
Why Our Code Smells
bkeepers
PRO
336
57k
Practical Orchestrator
shlominoach
189
11k
The Art of Programming - Codeland 2020
erikaheidi
54
13k
Large-scale JavaScript Application Architecture
addyosmani
512
110k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
Building Adaptive Systems
keathley
43
2.7k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
656
60k
Transcript
Making GitLab Faster
@yorickpeterse
[email protected]
GitLab Code, test, and deploy together
GitLab.com Statistics Users 544 306 Projects 955 340 Requests per
minute ~60 000 Storage ~30 TB
Response Timings Mean 350 ms 95th Percentile 1.23 s 99th
Percentile 3.17 s (backend only)
GitLab is slow, GitLab.com even more so
None
GitLab Problems
Limited production monitoring New Relic only on 1 out of
40 servers
No good development tools Only a collection of random Gems
No performance guides No guides on good/bad patterns, snippets to
use, workflow, etc
Lack of specialists Nobody was specifically focusing on performance
Badly Performing Code N+1 queries, inefficient algorithms, loading lots of
data into memory, etc
Solutions According to HackerNews
“Gitlab is one project that absolutely needs to be rebuilt
using Java. Both from a performance and a deployment perspective” https://news.ycombinator.com/item?id=11049717
“gogs was designed from the start to be fast. Nothing
but a rewrite would make GitLab as fast as gogs for basic stuff on small self-hosted servers.” https://news.ycombinator.com/item?id=11431410
Problem Solved! Let’s Rewrite GitLab in Java/Go/Haskell
Production Monitoring
Production monitoring is the first step to solving performance problems
GitLab’s Requirements 1. FOSS 2. Easy to set up 3.
Easy to query, visualize and add metrics 4. Low overhead 5. Support for percentiles
N-th percentile: N% of values are up to a given
value
“The 90th percentile is 1.5 seconds” means 90% completes in
1.5 seconds
GitLab Performance Monitoring
InfluxDB Time series DB with SQL-like query syntax
Grafana Time series visualization
Custom Ruby Code Used for measuring data, built from scratch
Transaction Metrics Rails, Grape, and Sidekiq
Code Region Metrics For measuring custom regions of code
Background Sampler Object counts, memory usage, GC statistics
None
None
Development Tools
Sherlock Development-only performance analysis, re-using the GitLab UI
1. Request details (URL, timings, etc) 2. Query timings, backtraces,
and EXPLAIN ANALYZE output 3. Line profiling
None
None
None
Documentation
GitLab Performance Guide Documents patterns to use/avoid, workflow, etc
http://docs.gitlab.com/ce/development/performance.html or https://is.gd/gitlab_webscale
Application Code
The language is rarely the problem
The database is not a black box
Rampant abuse of “pluck”
users.where(id: projects.pluck(:user_id))
users.where(id: projects.select(:user_id))
Combining Query Results
user.projects + public_projects + internal_projects
user.projects UNION public_projects UNION internal_projects
UNION not properly supported in ActiveRecord/Arel :<
Lots of Git Operations
if repo.exists? ... end
if repo.master_branch? foo end
Git Tag Existence https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3983
if all_tags.include?(ref_name) => if tag_exists?(ref_name)
Markdown References https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3969
for text in texts for reference in text process(reference)
process(texts)
1. Grab all references in a single pass 2. Query
any rows using the references 3. Replace text using the queried data
Calling “url_helpers” https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3486
include app.routes.url_helpers
include Gitlab::Routing.url_helpers
def self.url_helpers @url_helpers ||= Gitlab::Application. routes.url_helpers end
Recap
Language/framework is usually only a problem when the application is
already fully optimized.
Take advantage of your database’s features
There is no silver bullet to fixing performance problems, it
simply takes time, effort, and patience.
Keep an eye on GitLab 8.9 for many performance improvements
Questions?