Internals of GitLab and libgit2

Transcript

1. GitLab 与 libgit2 原理理
   Minqi Pan
   

   View Slide

2. I’m Minqi Pan
   github.com/pmq20
   twitter
   @psvr
   

   View Slide

3. GitLab 的励志故事
   • 2011年年：第⼀一⾏行行代码，基于 Rails 3
   • 2012年年：发布 2.1.0～4.0.0
   • 2013年年：升级到 Rails 4，发布 6.5.0

   在阿姆斯特丹丹注册公司 GitLab B.V.
   • 2014年年：申请 YC 孵化器成功，发布 7.7.0

   获得天使融资12万美⾦金金
   

   View Slide

4. GitLab 的励志故事
   • 2015年年1⽉月～3⽉月：进驻旧⾦金金⼭山湾区⼭山景城
   • 2015年年7⽉月：完成 150万 美元 pre-A 轮融资
   • 2015年年9⽉月：完成 570万 美元 A 轮融资
   • 2015年年10⽉月：发布 8.1.2
   

   View Slide

5. ⿊黑盒分析
   GitLab
   

   View Slide

6. ⿊黑盒分析
   GitLab
   HTTP
   80
   SSH
   22
   SMTP
   25/587
   

   View Slide

7. ⿊黑盒分析
   GitLab
   Redis
   Postgre
   SQL
   File
   System
   

   View Slide

8. ⽩白盒分析
   GitLab
   

   View Slide

9. ⽩白盒分析
   NGINX OpenSSH Server Postfix
   Unicorn Sidekiq
   Gitlab Workhorse
   libgit2
   gitlab-ci
   gitlab-rails gitlab-shell mattermost omnibus-ctl
   git
   

   View Slide

10. 具体场景分析
    • 创建项⽬目
    • 通过 HTTP 提交代码
    • 通过 SSH 克隆隆项⽬目
    • 在⻚页⾯面上查看提交历史
    

    View Slide

11. 创建项⽬目
    浏览器
    POST /projects
    ProjectsController#create
    CreateService#execute
    Project#save
    Postgre
    SQL
    NGINX
    Unicorn
    

    View Slide

12. 浏览器
    POST /projects
    …
    Gitlab::Shell#add_repository
    GitlabProjects#exec
    GitlabProjects#add_project
    $ git \
    --git-dir=/home/git/repositories/psvr/repo903.git \
    init \
    --bare
    $ /opt/gitlab/embedded/
    service/gitlab-shell/
    bin/gitlab-projects \
    add-project \
    psvr/repo903.git
    FileUtils.ln_sf(“/home/git/gitlab-shell/hooks",
    "/home/git/repositories/psvr/repo903.git/hooks")
    

    View Slide

13. 浏览器
    POST /projects
    CreateService#after_create_actions
    Gitlab::Shell#add_repository
    ProjectsController#create
    CreateService#execute
    Project#save
    浏览器
    302 重定向
    

    View Slide

14. 通过 HTTP 提交代码
    git push
    GET
    http://.../pmq2001/repo903.git/info/
    refs?service=git-receive-pack
    NGINX
    Gitlab Workhorse
    Unicorn
    func (h *gitHandler)
    doAuthRequest(r *http.Request)
    (result *http.Response, err error)
    GET http://.../pmq2001/repo903.git/
    info/refs?service=git-receive-pack
    去掉 body
    

    View Slide

15. 通过 HTTP 提交代码
    Unicorn
    git push …
    Grack::Auth#auth!
    Gitlab Workhorse
    validated the client request
    additional request metadata in JSON
    GET
    http://.../pmq2001/repo903.git/info/
    refs?service=git-receive-pack
    

    View Slide

16. 通过 HTTP 提交代码
    git push
    GET
    http://.../pmq2001/repo903.git/info/
    refs?service=git-receive-pack
    …
    Gitlab Workhorse
    $ git \
    receive-pack \
    --stateless-rpc \
    --advertise-refs \
    /home/git/repositories/psvr/repo903.git
    func handleGetInfoRefs
    (w http.ResponseWriter, r *gitRequest, _
    string)
    

    View Slide

17. 通过 HTTP 提交代码
    git push
    GET
    http://.../pmq2001/repo903.git/info/
    refs?service=git-receive-pack
    …
    Gitlab Workhorse
    

    View Slide

18. 通过 HTTP 提交代码
    git push
    POST
    http://.../pmq2001/repo903.git/git-
    receive-pack
    NGINX
    Gitlab Workhorse
    Unicorn
    func (h *gitHandler)
    doAuthRequest(r *http.Request)
    (result *http.Response, err error)
    POST /pmq2001/repo903.git/git-receive-
    pack
    去掉 body
    

    View Slide

19. 通过 HTTP 提交代码
    git push
    POST
    http://.../pmq2001/repo903.git/git-
    receive-pack
    NGINX
    Gitlab Workhorse
    func handlePostRPC
    (w http.ResponseWriter, r *gitRequest,
    rpc string)
    $ git \
    receive-pack \
    --stateless-rpc \
    /home/git/repositories/psvr/repo903.git
    

    View Slide

20. 通过 HTTP 提交代码
    git push
    POST
    http://.../pmq2001/repo903.git/git-
    receive-pack
    NGINX
    Gitlab Workhorse
    func handlePostRPC
    (w http.ResponseWriter, r *gitRequest,
    rpc string)
    $ git \
    receive-pack \
    --stateless-rpc \
    /home/git/repositories/psvr/repo903.git
    $ hooks/pre-receive
    gitlab-shell
    

    View Slide

21. 通过 HTTP 提交代码
    git push
    POST
    http://.../pmq2001/repo903.git/git-
    receive-pack
    $ hooks/pre-receive
    GitlabAccess#exec
    …
    Gitlab Workhorse
    Unicorn
    POST "/api/v3/internal/allowed"
    

    View Slide

22. 通过 HTTP 提交代码
    git push
    POST
    http://.../pmq2001/repo903.git/git-
    receive-pack
    …
    Unicorn
    $ hooks/post-receive
    Postgre
    SQL
    GitlabPostReceive#update_redis
    Sidekiq
    PostReceive
    

    View Slide

23. 通过 HTTP 提交代码
    git push
    POST
    http://.../pmq2001/repo903.git/git-
    receive-pack
    …
    Unicorn
    $ hooks/post-receive
    GitlabNet#broadcast_message
    GET /api/v3/internal/broadcast_message
    Unicorn
    ProjectCacheWorker
    Postgre
    SQL
    Sidekiq
    

    View Slide

24. 通过 HTTP 提交代码
    git push
    POST
    http://.../pmq2001/repo903.git/git-
    receive-pack
    NGINX
    Gitlab Workhorse
    

    View Slide

25. 通过 SSH 克隆隆项⽬目
    ssh git@gitlab.server OpenSSH Server
    gitlab-shell
    GitlabShell#exec
    

    View Slide

26. 通过 SSH 克隆隆项⽬目
    ssh git@gitlab.server
    GitlabShell#exec
    [root@i-jl15ph2j ~]# ssh git@gitlab.server
    Warning: Permanently added 'gitlab.server,1.2.3.4' (RSA) to the list of known hosts.
    PTY allocation request failed on channel 0
    Welcome to GitLab, psvr!
    Connection to gitlab.server closed.
    gitlab-shell
    OpenSSH Server
    

    View Slide

27. 通过 SSH 克隆隆项⽬目
    git clone
    GitlabShell#verify_access
    gitlab-shell
    OpenSSH Server
    git@gitlab.server:psvr/repo903.git
    Unicorn
    POST "/api/v3/internal/allowed"
    

    View Slide

28. 通过 SSH 克隆隆项⽬目
    git clone
    GitlabShell#verify_access
    gitlab-shell
    OpenSSH Server
    git@gitlab.server:psvr/repo903.git
    $ git-upload-pack \
    /var/opt/gitlab/git-data/repositories/psvr/repo903.git
    009bd9be3108e6e60b9954fec57f6ca00d4f74
    95170d HEADmulti_ack thin-pack side-
    band side-band-64k ofs-delta shallow
    no-progress include-tag
    multi_ack_detailed
    003fd9be3108e6e60b9954fec57f6ca00d4f74
    95170d refs/heads/master
    0000
    

    View Slide

29. 在⻚页⾯面上查看提交历史
    浏览器
    GET
    Projects::CommitsController#show
    Repository#commits
    NGINX
    Unicorn
    /psvr/repo903/commits/master
    gem - gitlab_git
    

    View Slide

30. 在⻚页⾯面上查看提交历史
    浏览器
    GET
    /psvr/repo903/commits/master
    gem - gitlab_git
    …
    Gitlab::Git::Repository#log
    $ git \
    --git-dir=home/git/repositories/psvr/repo903.git \
    -n 10 --format=%H --skip=100 --follow --no-merges
    

    View Slide

31. 在⻚页⾯面上查看提交历史
    浏览器
    GET
    /psvr/repo903/commits/master
    gem - gitlab_git
    …
    Gitlab::Git::Repository#log
    $ git \
    --git-dir=home/git/repositories/psvr/repo903.git \
    -n 10 --format=%H --skip=100 --follow --no-merges
    

    View Slide

32. 在⻚页⾯面上查看提交历史
    浏览器
    GET
    /psvr/repo903/commits/master
    gem - rugged
    …
    Rugged::Commit#message
    rb_git_commit_message_GET
    git_commit_message
    libgit2
    

    View Slide

33. libgit2 的特点
    portable, pure C, re-entrant, linkable, solid API
    提⽰示：对比 ruby 与 mruby
    

    View Slide

34. libgit2 是封装良好的
    • ⼤大部分头⽂文件位于 src/*.h，是不公开的
    • 像最基本的 git_repository 结构体都只有外部声明
    • include/git2/*.h 分类良好，快速定位接⼝口，包含
    丰富的注释（对比 mruby =.=）
    • [ demo ]
    

    View Slide

35. libgit2 是抽象良好的
    • git2/sys/*_backend.h 解耦, 但是 index 没有解耦
    • 通过这些函数挂接新的后端
    

    View Slide

36. View Slide

37. Thank you
    https://github.com/pmq20/
    

    View Slide