Making GitLab Faster

Talk given at RubyC 2016

Transcript

1. Making GitLab Faster
2. @yorickpeterse yorickpeterse@gmail.com
3. GitLab Code, test, and deploy together
4. GitLab.com Statistics Users 544 306 Projects 955 340 Requests per minute ~60 000 Storage ~30 TB
5. Response Timings Mean 350 ms 95th Percentile 1.23 s 99th Percentile 3.17 s (backend only)
6. GitLab is slow, GitLab.com even more so
8. GitLab Problems
9. Limited production monitoring New Relic only on 1 out of 40 servers
10. No good development tools Only a collection of random Gems
11. No performance guides No guides on good/bad patterns, snippets to use, workflow, etc
12. Lack of specialists Nobody was specifically focusing on performance
13. Badly Performing Code N+1 queries, inefficient algorithms, loading lots of data into memory, etc
14. Solutions According to HackerNews
15. “Gitlab is one project that absolutely needs to be rebuilt using Java. Both from a performance and a deployment perspective” https://news.ycombinator.com/item?id=11049717
16. “gogs was designed from the start to be fast. Nothing but a rewrite would make GitLab as fast as gogs for basic stuff on small self-hosted servers.” https://news.ycombinator.com/item?id=11431410
17. Problem Solved! Let’s Rewrite GitLab in Java/Go/Haskell
18. Production Monitoring
19. Production monitoring is the first step to solving performance problems
20. GitLab’s Requirements 1. FOSS 2. Easy to set up 3. Easy to query, visualize and add metrics 4. Low overhead 5. Support for percentiles
21. N-th percentile: N% of values are up to a given value
22. “The 90th percentile is 1.5 seconds” means 90% completes in 1.5 seconds
23. GitLab Performance Monitoring
24. InfluxDB Time series DB with SQL-like query syntax
25. Grafana Time series visualization
26. Custom Ruby Code Used for measuring data, built from scratch
27. Transaction Metrics Rails, Grape, and Sidekiq
28. Code Region Metrics For measuring custom regions of code
29. Background Sampler Object counts, memory usage, GC statistics
32. Development Tools
33. Sherlock Development-only performance analysis, re-using the GitLab UI
34. 1. Request details (URL, timings, etc) 2. Query timings, backtraces, and EXPLAIN ANALYZE output 3. Line profiling
38. Documentation
39. GitLab Performance Guide Documents patterns to use/avoid, workflow, etc
40. http://docs.gitlab.com/ce/development/performance.html or https://is.gd/gitlab_webscale
41. Application Code
42. The language is rarely the problem
43. The database is not a black box
44. Rampant abuse of “pluck”
45. users.where(id: projects.pluck(:user_id))
46. users.where(id: projects.select(:user_id))
47. Combining Query Results
48. user.projects + public_projects + internal_projects
49. user.projects UNION public_projects UNION internal_projects
50. UNION not properly supported in ActiveRecord/Arel :<
51. Lots of Git Operations
52. if repo.exists? ... end
53. if repo.master_branch? foo end
54. Git Tag Existence https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3983
55. if all_tags.include?(ref_name) => if tag_exists?(ref_name)
56. Markdown References https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3969
57. for text in texts for reference in text process(reference)
58. process(texts)
59. 1. Grab all references in a single pass 2. Query any rows using the references 3. Replace text using the queried data
60. Calling “url_helpers” https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3486
61. include app.routes.url_helpers
62. include Gitlab::Routing.url_helpers
63. def self.url_helpers @url_helpers ||= Gitlab::Application. routes.url_helpers end
64. Recap
65. Language/framework is usually only a problem when the application is already fully optimized.
66. Take advantage of your database’s features
67. There is no silver bullet to fixing performance problems, it simply takes time, effort, and patience.
68. Keep an eye on GitLab 8.9 for many performance improvements
69. Questions?