Internals of GitLab and libgit2

Slide 1

Slide 1 text

GitLab 与 libgit2 原理理 Minqi Pan

Slide 2

Slide 2 text

I’m Minqi Pan github.com/pmq20 twitter @psvr

Slide 3

Slide 3 text

GitLab 的励志故事 • 2011年年：第⼀一⾏行行代码，基于 Rails 3 • 2012年年：发布 2.1.0～4.0.0 • 2013年年：升级到 Rails 4，发布 6.5.0  在阿姆斯特丹丹注册公司 GitLab B.V. • 2014年年：申请 YC 孵化器成功，发布 7.7.0  获得天使融资12万美⾦金金

Slide 4

Slide 4 text

GitLab 的励志故事 • 2015年年1⽉月～3⽉月：进驻旧⾦金金⼭山湾区⼭山景城 • 2015年年7⽉月：完成 150万美元 pre-A 轮融资 • 2015年年9⽉月：完成 570万美元 A 轮融资 • 2015年年10⽉月：发布 8.1.2

Slide 5

Slide 5 text

⿊黑盒分析 GitLab

Slide 6

Slide 6 text

⿊黑盒分析 GitLab HTTP 80 SSH 22 SMTP 25/587

Slide 7

Slide 7 text

⿊黑盒分析 GitLab Redis Postgre SQL File System

Slide 8

Slide 8 text

⽩白盒分析 GitLab

Slide 9

Slide 9 text

⽩白盒分析 NGINX OpenSSH Server Postﬁx Unicorn Sidekiq Gitlab Workhorse libgit2 gitlab-ci gitlab-rails gitlab-shell mattermost omnibus-ctl git

Slide 10

Slide 10 text

具体场景分析 • 创建项⽬目 • 通过 HTTP 提交代码 • 通过 SSH 克隆隆项⽬目 • 在⻚页⾯面上查看提交历史

Slide 11

Slide 11 text

创建项⽬目浏览器 POST /projects ProjectsController#create CreateService#execute Project#save Postgre SQL NGINX Unicorn

Slide 12

Slide 12 text

浏览器 POST /projects … Gitlab::Shell#add_repository GitlabProjects#exec GitlabProjects#add_project $ git \ --git-dir=/home/git/repositories/psvr/repo903.git \ init \ --bare $ /opt/gitlab/embedded/ service/gitlab-shell/ bin/gitlab-projects \ add-project \ psvr/repo903.git FileUtils.ln_sf(“/home/git/gitlab-shell/hooks", "/home/git/repositories/psvr/repo903.git/hooks")

Slide 13

Slide 13 text

浏览器 POST /projects CreateService#after_create_actions Gitlab::Shell#add_repository ProjectsController#create CreateService#execute Project#save 浏览器 302 重定向

Slide 14

Slide 14 text

通过 HTTP 提交代码 git push GET http://.../pmq2001/repo903.git/info/ refs?service=git-receive-pack NGINX Gitlab Workhorse Unicorn func (h *gitHandler) doAuthRequest(r *http.Request) (result *http.Response, err error) GET http://.../pmq2001/repo903.git/ info/refs?service=git-receive-pack 去掉 body

Slide 15

Slide 15 text

通过 HTTP 提交代码 Unicorn git push … Grack::Auth#auth! Gitlab Workhorse validated the client request additional request metadata in JSON GET http://.../pmq2001/repo903.git/info/ refs?service=git-receive-pack

Slide 16

Slide 16 text

通过 HTTP 提交代码 git push GET http://.../pmq2001/repo903.git/info/ refs?service=git-receive-pack … Gitlab Workhorse $ git \ receive-pack \ --stateless-rpc \ --advertise-refs \ /home/git/repositories/psvr/repo903.git func handleGetInfoRefs (w http.ResponseWriter, r *gitRequest, _ string)

Slide 17

Slide 17 text

通过 HTTP 提交代码 git push GET http://.../pmq2001/repo903.git/info/ refs?service=git-receive-pack … Gitlab Workhorse

Slide 18

Slide 18 text

通过 HTTP 提交代码 git push POST http://.../pmq2001/repo903.git/git- receive-pack NGINX Gitlab Workhorse Unicorn func (h *gitHandler) doAuthRequest(r *http.Request) (result *http.Response, err error) POST /pmq2001/repo903.git/git-receive- pack 去掉 body

Slide 19

Slide 19 text

通过 HTTP 提交代码 git push POST http://.../pmq2001/repo903.git/git- receive-pack NGINX Gitlab Workhorse func handlePostRPC (w http.ResponseWriter, r *gitRequest, rpc string) $ git \ receive-pack \ --stateless-rpc \ /home/git/repositories/psvr/repo903.git

Slide 20

Slide 20 text

Slide 21

Slide 21 text

通过 HTTP 提交代码 git push POST http://.../pmq2001/repo903.git/git- receive-pack $ hooks/pre-receive GitlabAccess#exec … Gitlab Workhorse Unicorn POST "/api/v3/internal/allowed"

Slide 22

Slide 22 text

通过 HTTP 提交代码 git push POST http://.../pmq2001/repo903.git/git- receive-pack … Unicorn $ hooks/post-receive Postgre SQL GitlabPostReceive#update_redis Sidekiq PostReceive

Slide 23

Slide 23 text

通过 HTTP 提交代码 git push POST http://.../pmq2001/repo903.git/git- receive-pack … Unicorn $ hooks/post-receive GitlabNet#broadcast_message GET /api/v3/internal/broadcast_message Unicorn ProjectCacheWorker Postgre SQL Sidekiq

Slide 24

Slide 24 text

通过 HTTP 提交代码 git push POST http://.../pmq2001/repo903.git/git- receive-pack NGINX Gitlab Workhorse

Slide 25

Slide 25 text

通过 SSH 克隆隆项⽬目 ssh [email protected] OpenSSH Server gitlab-shell GitlabShell#exec

Slide 26

Slide 26 text

通过 SSH 克隆隆项⽬目 ssh [email protected] GitlabShell#exec [root@i-jl15ph2j ~]# ssh [email protected] Warning: Permanently added 'gitlab.server,1.2.3.4' (RSA) to the list of known hosts. PTY allocation request failed on channel 0 Welcome to GitLab, psvr! Connection to gitlab.server closed. gitlab-shell OpenSSH Server

Slide 27

Slide 27 text

通过 SSH 克隆隆项⽬目 git clone GitlabShell#verify_access gitlab-shell OpenSSH Server [email protected]:psvr/repo903.git Unicorn POST "/api/v3/internal/allowed"

Slide 28

Slide 28 text

通过 SSH 克隆隆项⽬目 git clone GitlabShell#verify_access gitlab-shell OpenSSH Server [email protected]:psvr/repo903.git $ git-upload-pack \ /var/opt/gitlab/git-data/repositories/psvr/repo903.git 009bd9be3108e6e60b9954fec57f6ca00d4f74 95170d HEADmulti_ack thin-pack side- band side-band-64k ofs-delta shallow no-progress include-tag multi_ack_detailed 003fd9be3108e6e60b9954fec57f6ca00d4f74 95170d refs/heads/master 0000

Slide 29

Slide 29 text

在⻚页⾯面上查看提交历史浏览器 GET Projects::CommitsController#show Repository#commits NGINX Unicorn /psvr/repo903/commits/master gem - gitlab_git

Slide 30

Slide 30 text

在⻚页⾯面上查看提交历史浏览器 GET /psvr/repo903/commits/master gem - gitlab_git … Gitlab::Git::Repository#log $ git \ --git-dir=home/git/repositories/psvr/repo903.git \ -n 10 --format=%H --skip=100 --follow --no-merges

Slide 31

Slide 31 text

Slide 32

Slide 32 text

在⻚页⾯面上查看提交历史浏览器 GET /psvr/repo903/commits/master gem - rugged … Rugged::Commit#message rb_git_commit_message_GET git_commit_message libgit2

Slide 33

Slide 33 text

libgit2 的特点 portable, pure C, re-entrant, linkable, solid API 提⽰示：对比 ruby 与 mruby

Slide 34

Slide 34 text

libgit2 是封装良好的 • ⼤大部分头⽂文件位于 src/*.h，是不公开的 • 像最基本的 git_repository 结构体都只有外部声明 • include/git2/*.h 分类良好，快速定位接⼝口，包含丰富的注释（对比 mruby =.=） • [ demo ]

Slide 35

Slide 35 text

libgit2 是抽象良好的 • git2/sys/*_backend.h 解耦, 但是 index 没有解耦 • 通过这些函数挂接新的后端