$30 off During Our Annual Pro Sale. View Details »

Internals of GitLab and libgit2

Minqi Pan
November 03, 2015

Internals of GitLab and libgit2

Minqi Pan

November 03, 2015
Tweet

More Decks by Minqi Pan

Other Decks in Programming

Transcript

  1. GitLab 与 libgit2 原理理
    Minqi Pan

    View Slide

  2. I’m Minqi Pan
    github.com/pmq20
    twitter
    @psvr

    View Slide

  3. GitLab 的励志故事
    • 2011年年:第⼀一⾏行行代码,基于 Rails 3
    • 2012年年:发布 2.1.0~4.0.0
    • 2013年年:升级到 Rails 4,发布 6.5.0

    在阿姆斯特丹丹注册公司 GitLab B.V.
    • 2014年年:申请 YC 孵化器成功,发布 7.7.0

    获得天使融资12万美⾦金金

    View Slide

  4. GitLab 的励志故事
    • 2015年年1⽉月~3⽉月:进驻旧⾦金金⼭山湾区⼭山景城
    • 2015年年7⽉月:完成 150万 美元 pre-A 轮融资
    • 2015年年9⽉月:完成 570万 美元 A 轮融资
    • 2015年年10⽉月:发布 8.1.2

    View Slide

  5. ⿊黑盒分析
    GitLab

    View Slide

  6. ⿊黑盒分析
    GitLab
    HTTP
    80
    SSH
    22
    SMTP
    25/587

    View Slide

  7. ⿊黑盒分析
    GitLab
    Redis
    Postgre
    SQL
    File
    System

    View Slide

  8. ⽩白盒分析
    GitLab

    View Slide

  9. ⽩白盒分析
    NGINX OpenSSH Server Postfix
    Unicorn Sidekiq
    Gitlab Workhorse
    libgit2
    gitlab-ci
    gitlab-rails gitlab-shell mattermost omnibus-ctl
    git

    View Slide

  10. 具体场景分析
    • 创建项⽬目
    • 通过 HTTP 提交代码
    • 通过 SSH 克隆隆项⽬目
    • 在⻚页⾯面上查看提交历史

    View Slide

  11. 创建项⽬目
    浏览器
    POST /projects
    ProjectsController#create
    CreateService#execute
    Project#save
    Postgre
    SQL
    NGINX
    Unicorn

    View Slide

  12. 浏览器
    POST /projects

    Gitlab::Shell#add_repository
    GitlabProjects#exec
    GitlabProjects#add_project
    $ git \
    --git-dir=/home/git/repositories/psvr/repo903.git \
    init \
    --bare
    $ /opt/gitlab/embedded/
    service/gitlab-shell/
    bin/gitlab-projects \
    add-project \
    psvr/repo903.git
    FileUtils.ln_sf(“/home/git/gitlab-shell/hooks",
    "/home/git/repositories/psvr/repo903.git/hooks")

    View Slide

  13. 浏览器
    POST /projects
    CreateService#after_create_actions
    Gitlab::Shell#add_repository
    ProjectsController#create
    CreateService#execute
    Project#save
    浏览器
    302 重定向

    View Slide

  14. 通过 HTTP 提交代码
    git push
    GET
    http://.../pmq2001/repo903.git/info/
    refs?service=git-receive-pack
    NGINX
    Gitlab Workhorse
    Unicorn
    func (h *gitHandler)
    doAuthRequest(r *http.Request)
    (result *http.Response, err error)
    GET http://.../pmq2001/repo903.git/
    info/refs?service=git-receive-pack
    去掉 body

    View Slide

  15. 通过 HTTP 提交代码
    Unicorn
    git push …
    Grack::Auth#auth!
    Gitlab Workhorse
    validated the client request
    additional request metadata in JSON
    GET
    http://.../pmq2001/repo903.git/info/
    refs?service=git-receive-pack

    View Slide

  16. 通过 HTTP 提交代码
    git push
    GET
    http://.../pmq2001/repo903.git/info/
    refs?service=git-receive-pack

    Gitlab Workhorse
    $ git \
    receive-pack \
    --stateless-rpc \
    --advertise-refs \
    /home/git/repositories/psvr/repo903.git
    func handleGetInfoRefs
    (w http.ResponseWriter, r *gitRequest, _
    string)

    View Slide

  17. 通过 HTTP 提交代码
    git push
    GET
    http://.../pmq2001/repo903.git/info/
    refs?service=git-receive-pack

    Gitlab Workhorse

    View Slide

  18. 通过 HTTP 提交代码
    git push
    POST
    http://.../pmq2001/repo903.git/git-
    receive-pack
    NGINX
    Gitlab Workhorse
    Unicorn
    func (h *gitHandler)
    doAuthRequest(r *http.Request)
    (result *http.Response, err error)
    POST /pmq2001/repo903.git/git-receive-
    pack
    去掉 body

    View Slide

  19. 通过 HTTP 提交代码
    git push
    POST
    http://.../pmq2001/repo903.git/git-
    receive-pack
    NGINX
    Gitlab Workhorse
    func handlePostRPC
    (w http.ResponseWriter, r *gitRequest,
    rpc string)
    $ git \
    receive-pack \
    --stateless-rpc \
    /home/git/repositories/psvr/repo903.git

    View Slide

  20. 通过 HTTP 提交代码
    git push
    POST
    http://.../pmq2001/repo903.git/git-
    receive-pack
    NGINX
    Gitlab Workhorse
    func handlePostRPC
    (w http.ResponseWriter, r *gitRequest,
    rpc string)
    $ git \
    receive-pack \
    --stateless-rpc \
    /home/git/repositories/psvr/repo903.git
    $ hooks/pre-receive
    gitlab-shell

    View Slide

  21. 通过 HTTP 提交代码
    git push
    POST
    http://.../pmq2001/repo903.git/git-
    receive-pack
    $ hooks/pre-receive
    GitlabAccess#exec

    Gitlab Workhorse
    Unicorn
    POST "/api/v3/internal/allowed"

    View Slide

  22. 通过 HTTP 提交代码
    git push
    POST
    http://.../pmq2001/repo903.git/git-
    receive-pack

    Unicorn
    $ hooks/post-receive
    Postgre
    SQL
    GitlabPostReceive#update_redis
    Sidekiq
    PostReceive

    View Slide

  23. 通过 HTTP 提交代码
    git push
    POST
    http://.../pmq2001/repo903.git/git-
    receive-pack

    Unicorn
    $ hooks/post-receive
    GitlabNet#broadcast_message
    GET /api/v3/internal/broadcast_message
    Unicorn
    ProjectCacheWorker
    Postgre
    SQL
    Sidekiq

    View Slide

  24. 通过 HTTP 提交代码
    git push
    POST
    http://.../pmq2001/repo903.git/git-
    receive-pack
    NGINX
    Gitlab Workhorse

    View Slide

  25. 通过 SSH 克隆隆项⽬目
    ssh [email protected] OpenSSH Server
    gitlab-shell
    GitlabShell#exec

    View Slide

  26. 通过 SSH 克隆隆项⽬目
    ssh [email protected]
    GitlabShell#exec
    [root@i-jl15ph2j ~]# ssh [email protected]
    Warning: Permanently added 'gitlab.server,1.2.3.4' (RSA) to the list of known hosts.
    PTY allocation request failed on channel 0
    Welcome to GitLab, psvr!
    Connection to gitlab.server closed.
    gitlab-shell
    OpenSSH Server

    View Slide

  27. 通过 SSH 克隆隆项⽬目
    git clone
    GitlabShell#verify_access
    gitlab-shell
    OpenSSH Server
    [email protected]:psvr/repo903.git
    Unicorn
    POST "/api/v3/internal/allowed"

    View Slide

  28. 通过 SSH 克隆隆项⽬目
    git clone
    GitlabShell#verify_access
    gitlab-shell
    OpenSSH Server
    [email protected]:psvr/repo903.git
    $ git-upload-pack \
    /var/opt/gitlab/git-data/repositories/psvr/repo903.git
    009bd9be3108e6e60b9954fec57f6ca00d4f74
    95170d HEADmulti_ack thin-pack side-
    band side-band-64k ofs-delta shallow
    no-progress include-tag
    multi_ack_detailed
    003fd9be3108e6e60b9954fec57f6ca00d4f74
    95170d refs/heads/master
    0000

    View Slide

  29. 在⻚页⾯面上查看提交历史
    浏览器
    GET
    Projects::CommitsController#show
    Repository#commits
    NGINX
    Unicorn
    /psvr/repo903/commits/master
    gem - gitlab_git

    View Slide

  30. 在⻚页⾯面上查看提交历史
    浏览器
    GET
    /psvr/repo903/commits/master
    gem - gitlab_git

    Gitlab::Git::Repository#log
    $ git \
    --git-dir=home/git/repositories/psvr/repo903.git \
    -n 10 --format=%H --skip=100 --follow --no-merges

    View Slide

  31. 在⻚页⾯面上查看提交历史
    浏览器
    GET
    /psvr/repo903/commits/master
    gem - gitlab_git

    Gitlab::Git::Repository#log
    $ git \
    --git-dir=home/git/repositories/psvr/repo903.git \
    -n 10 --format=%H --skip=100 --follow --no-merges

    View Slide

  32. 在⻚页⾯面上查看提交历史
    浏览器
    GET
    /psvr/repo903/commits/master
    gem - rugged

    Rugged::Commit#message
    rb_git_commit_message_GET
    git_commit_message
    libgit2

    View Slide

  33. libgit2 的特点
    portable, pure C, re-entrant, linkable, solid API
    提⽰示:对比 ruby 与 mruby

    View Slide

  34. libgit2 是封装良好的
    • ⼤大部分头⽂文件位于 src/*.h,是不公开的
    • 像最基本的 git_repository 结构体都只有外部声明
    • include/git2/*.h 分类良好,快速定位接⼝口,包含
    丰富的注释(对比 mruby =.=)
    • [ demo ]

    View Slide

  35. libgit2 是抽象良好的
    • git2/sys/*_backend.h 解耦, 但是 index 没有解耦
    • 通过这些函数挂接新的后端

    View Slide

  36. View Slide

  37. Thank you
    https://github.com/pmq20/

    View Slide