Building a git reverse proxy in Go for Caddy

Scaling non-trivial software can be complex. When your core business depends on custom communication protocols, you may end up having to get your hands dirty and write custom system services. In this talk I’m going to show how we can leverage the modular Caddy HTTP server and implement a Git reverse proxy bundled in it. You may also learn a few things about Git internals you never had the chance to ask anyone about.

Links at the end of the presentation:
The Boiler Plate: https://gitlab.com/brodock/caddy-helloworld
The SMART HTTP Proxy: https://gitlab.com/brodock/caddy-gitproxy

Transcript

1. View Slide

2. Building a git
   reverse proxy
   G f C
   

   View Slide

3. @brodock
   blog.gabrielmazetto.eti.br
   Gabriel Mazetto
   

   View Slide

4. View Slide

5. That version control thing...
   

   View Slide

6. This is a talk about git
   and its internals
   f G ...
   

   View Slide

7. I WILL NOT TEACH YOU
   HOW TO
   

   View Slide

8. View Slide

9. I have to introduce you to a
   lot of concepts before...
   

   View Slide

10. $ git help
    clone Clone a repository into a new directory
    init Create an empty Git repository or reinitialize an existing one
    add Add file contents to the index
    mv Move or rename a file, a directory, or a symlink
    reset Reset current HEAD to the specified state
    rm Remove files from the working tree and from the index
    log Show commit logs
    show Show various types of objects
    status Show the working tree status
    branch List, create, or delete branches
    checkout Switch branches or restore working tree files
    commit Record changes to the repository
    diff Show changes between commits, commit and working tree, etc
    merge Join two or more development histories together
    rebase Forward-port local commits to the updated upstream head
    tag Create, list, delete or verify a tag object signed with GPG
    fetch Download objects and refs from another repository
    pull Fetch from and integrate with another repository or a local branch
    push Update remote refs along with associated objects
    

    View Slide

11. What git really is?
    

    View Slide

12. $ find bin/ libexec/ | wc -l
    76
    not a single binary file...
    

    View Slide

13. View Slide

14. Unix Philosophy:
    S , ,
    

    View Slide

15. $ find bin/ libexec/
    ./bin/git
    ./bin/git-cvsserver
    ./bin/git-receive-pack
    ./bin/git-shell
    ./bin/git-upload-archive
    ./bin/git-upload-pack
    ./libexec/gitcore/git-rebase
    ./libexec/gitcore/git-stash
    ./libexec/gitcore/git-submodule
    ./libexec/gitcore/git-mergetool
    ./libexec/gitcore/git-bisect
    ./libexec/gitcore/git-daemon
    ./libexec/gitcore/git-rebase--am
    ./libexec/gitcore/git-parse-remote
    ./libexec/gitcore/git-svn
    ./libexec/gitcore/git-http-backendu
    ...
    perl
    shell script
    

    View Slide

16. Where is my code ?
    

    View Slide

17. P
    Object Database (ODB)
    

    View Slide

18. How current ODB backend
    work?
    

    View Slide

19. $ git log
    commit 74ac5c498323b39dfb14eec9244b5f628b717555
    Author: Gabriel Mazetto
    Date: Mon Sep 18 10:49:51 2017 +0200
    Added support for Git RPC with upload-pack (clone works now)
    type
    SHA1 hash (key)
    content (data)
    

    View Slide

20. So is this how the commit
    object look like?
    

    View Slide

21. $ git show-ref --head --hash HEAD
    74ac5c498323b39dfb14eec9244b5f628b717555
    $ git cat-file commit HEAD
    tree f7530ab4de2105b49789dda745fb450b95c2a33b
    parent 2d3df47cef64497c77f52f0803e2ca1467f5be1a
    author Gabriel Mazetto 1505724591 +0200
    committer Gabriel Mazetto 1505724591 +0200
    Added support for Git RPC with upload-pack (clone works now)
    

    View Slide

22. $ git cat-file -p f7530ab4de2105b49789dda745fb450b95c2a33b
    100644 blob ffd2a4a2e1b6f84ec571a84a4bd6f001643a0745 .gitignore
    100644 blob 0929ab8179725c680176ea5001dbab28faea166d README.md
    100644 blob 1cdeb2b8f92771a60fe7152f541fdb939f185b86 git.go
    100644 blob 7ddbc56e43e5b6ab8e7e00c88589f78c2b7eef04 gitproxy.go
    100644 blob 76133a6764e0e49b787dbb49a3103f64f83246f1 setup.go
    100644 blob 18b3b795d9a196ecfaf9548be5f721f3f878d9ae setup_test.go
    

    View Slide

23. $ git show-ref --head --hash HEAD
    74ac5c498323b39dfb14eec9244b5f628b717555
    $ git cat-file commit HEAD
    tree f7530ab4de2105b49789dda745fb450b95c2a33b
    parent 2d3df47cef64497c77f52f0803e2ca1467f5be1a
    author Gabriel Mazetto 1505724591 +0200
    committer Gabriel Mazetto 1505724591 +0200
    Added support for Git RPC with upload-pack (clone works now)
    

    View Slide

24. $ git cat-file 2d3df47cef64497c77f52f0803e2ca1467f5be1a
    tree 615b472d533b66b908f7d79a0b9d18be32779de4
    parent 7fa2356cc5ecd80e949803447dfeb8b566764186
    author Gabriel Mazetto 1505720787 +0200
    committer Gabriel Mazetto 1505721333 +0200
    Added GitAdapter, fixed things and now ls-remote works.
    

    View Slide

25. 74ac5c4
    commit
    f7530ab
    tree
    ffd2a4a
    blob
    1cdeb2b
    blob
    2d3df47
    commit
    615b472
    tree eb5bce0
    blob
    ...
    

    View Slide

26. How is the hash
    generated?
    

    View Slide

27. SHA1([obj-type][size][NUL][object-content])
    

    View Slide

28. $ printf "commit %s\0" $(git cat-file
    commit HEAD | wc -c)
    commit 279
    NUL byte character
    

    View Slide

29. $ (printf "commit %s\0" $(git cat-file commit HEAD | wc -c);
    git cat-file commit HEAD)
    commit 279tree f7530ab4de2105b49789dda745fb450b95c2a33b
    parent 2d3df47cef64497c77f52f0803e2ca1467f5be1a
    author Gabriel Mazetto 1505724591 +0200
    committer Gabriel Mazetto 1505724591 +0200
    Added support for Git RPC with upload-pack (clone works now)
    

    View Slide

30. $ (printf "commit %s\0" $(git cat-file commit HEAD | wc -c);
    git cat-file commit HEAD) | sha1sum
    74ac5c498323b39dfb14eec9244b5f628b717555
    $ git show-ref --head --hash HEAD
    74ac5c498323b39dfb14eec9244b5f628b717555
    

    View Slide

31. View Slide

32. Let's talk about protocols...
    

    View Slide

33. Syncing your
    

    View Slide

34. git, ssh, https*
    WebDAV, dumb, smart
    

    View Slide

35. The git:// protocol
    $ git daemon --base-path=/srv/git
    READONLY
    

    View Slide

36. The ssh:// protocol
    $ git-upload-pack
    $ git-receive-pack
    

    View Slide

37. The webdav:// protocol
    

    View Slide

38. The dumb protocol
    $ update-server-info
    READONLY
    

    View Slide

39. The smart protocol
    $ git-upload-pack
    $ git-receive-pack
    

    View Slide

40. Let's talk about
    C
    

    View Slide

41. Single Binary
    $ caddy -conf /path/to/Caddyfile
    

    View Slide

42. It's not only a
    HTTP
    

    View Slide

43. Secure by Default!
    L ' E
    MITM attack detection
    

    View Slide

44. label1 {
    directive1 arg1
    directive2 arg1 arg2
    directive3 {
    subdir1 arg1 arg2
    subdir2 arg1 arg2
    }
    }
    

    View Slide

45. https://awesomeapp.example.com {
    root /srv/awesomeapp.example.com
    log /var/opt/log/awesomeapp.log {
    rotate_size 50 # after 50MB
    rotate_keep 20 # log files to keep
    rotate_compress # gzip after rotation
    }
    proxy / localhost:3000 localhost:3001 {
    policy round_robin # rotate requests
    transparent # forward request IPs
    }
    }
    

    View Slide

46. Now, try to do the same
    config with nginx...
    

    View Slide

47. View Slide

48. Let's try to build some
    f ...
    

    View Slide

49. There are few types of
    plugins
    

    View Slide

50. We are going to extend the
    Caddyfile and add a HTTP
    Middleware
    

    View Slide

51. The Boilerplate
    https://gitlab.com/brodock/caddy-helloworld
    

    View Slide

52. The SMART HTTP Proxy
    https://gitlab.com/brodock/caddy-gitproxy
    

    View Slide

53. Q ?
    

    View Slide