Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building a git reverse proxy in Go for Caddy

Building a git reverse proxy in Go for Caddy

Scaling non-trivial software can be complex. When your core business depends on custom communication protocols, you may end up having to get your hands dirty and write custom system services. In this talk I’m going to show how we can leverage the modular Caddy HTTP server and implement a Git reverse proxy bundled in it. You may also learn a few things about Git internals you never had the chance to ask anyone about.

Links at the end of the presentation:
The Boiler Plate: https://gitlab.com/brodock/caddy-helloworld
The SMART HTTP Proxy: https://gitlab.com/brodock/caddy-gitproxy

Gabriel Mazetto

September 30, 2017
Tweet

More Decks by Gabriel Mazetto

Other Decks in Technology

Transcript

  1. View Slide

  2. Building a git
    reverse proxy
    G f C

    View Slide

  3. @brodock
    blog.gabrielmazetto.eti.br
    Gabriel Mazetto

    View Slide

  4. View Slide

  5. That version control thing...

    View Slide

  6. This is a talk about git
    and its internals
    f G ...

    View Slide

  7. I WILL NOT TEACH YOU
    HOW TO

    View Slide

  8. View Slide

  9. I have to introduce you to a
    lot of concepts before...

    View Slide

  10. $ git help
    clone Clone a repository into a new directory
    init Create an empty Git repository or reinitialize an existing one
    add Add file contents to the index
    mv Move or rename a file, a directory, or a symlink
    reset Reset current HEAD to the specified state
    rm Remove files from the working tree and from the index
    log Show commit logs
    show Show various types of objects
    status Show the working tree status
    branch List, create, or delete branches
    checkout Switch branches or restore working tree files
    commit Record changes to the repository
    diff Show changes between commits, commit and working tree, etc
    merge Join two or more development histories together
    rebase Forward-port local commits to the updated upstream head
    tag Create, list, delete or verify a tag object signed with GPG
    fetch Download objects and refs from another repository
    pull Fetch from and integrate with another repository or a local branch
    push Update remote refs along with associated objects

    View Slide

  11. What git really is?

    View Slide

  12. $ find bin/ libexec/ | wc -l
    76
    not a single binary file...

    View Slide

  13. View Slide

  14. Unix Philosophy:
    S , ,

    View Slide

  15. $ find bin/ libexec/
    ./bin/git
    ./bin/git-cvsserver
    ./bin/git-receive-pack
    ./bin/git-shell
    ./bin/git-upload-archive
    ./bin/git-upload-pack
    ./libexec/gitcore/git-rebase
    ./libexec/gitcore/git-stash
    ./libexec/gitcore/git-submodule
    ./libexec/gitcore/git-mergetool
    ./libexec/gitcore/git-bisect
    ./libexec/gitcore/git-daemon
    ./libexec/gitcore/git-rebase--am
    ./libexec/gitcore/git-parse-remote
    ./libexec/gitcore/git-svn
    ./libexec/gitcore/git-http-backendu
    ...
    perl
    shell script

    View Slide

  16. Where is my code ?

    View Slide

  17. P
    Object Database (ODB)

    View Slide

  18. How current ODB backend
    work?

    View Slide

  19. $ git log
    commit 74ac5c498323b39dfb14eec9244b5f628b717555
    Author: Gabriel Mazetto
    Date: Mon Sep 18 10:49:51 2017 +0200
    Added support for Git RPC with upload-pack (clone works now)
    type
    SHA1 hash (key)
    content (data)

    View Slide

  20. So is this how the commit
    object look like?

    View Slide

  21. $ git show-ref --head --hash HEAD
    74ac5c498323b39dfb14eec9244b5f628b717555
    $ git cat-file commit HEAD
    tree f7530ab4de2105b49789dda745fb450b95c2a33b
    parent 2d3df47cef64497c77f52f0803e2ca1467f5be1a
    author Gabriel Mazetto 1505724591 +0200
    committer Gabriel Mazetto 1505724591 +0200
    Added support for Git RPC with upload-pack (clone works now)

    View Slide

  22. $ git cat-file -p f7530ab4de2105b49789dda745fb450b95c2a33b
    100644 blob ffd2a4a2e1b6f84ec571a84a4bd6f001643a0745 .gitignore
    100644 blob 0929ab8179725c680176ea5001dbab28faea166d README.md
    100644 blob 1cdeb2b8f92771a60fe7152f541fdb939f185b86 git.go
    100644 blob 7ddbc56e43e5b6ab8e7e00c88589f78c2b7eef04 gitproxy.go
    100644 blob 76133a6764e0e49b787dbb49a3103f64f83246f1 setup.go
    100644 blob 18b3b795d9a196ecfaf9548be5f721f3f878d9ae setup_test.go

    View Slide

  23. $ git show-ref --head --hash HEAD
    74ac5c498323b39dfb14eec9244b5f628b717555
    $ git cat-file commit HEAD
    tree f7530ab4de2105b49789dda745fb450b95c2a33b
    parent 2d3df47cef64497c77f52f0803e2ca1467f5be1a
    author Gabriel Mazetto 1505724591 +0200
    committer Gabriel Mazetto 1505724591 +0200
    Added support for Git RPC with upload-pack (clone works now)

    View Slide

  24. $ git cat-file 2d3df47cef64497c77f52f0803e2ca1467f5be1a
    tree 615b472d533b66b908f7d79a0b9d18be32779de4
    parent 7fa2356cc5ecd80e949803447dfeb8b566764186
    author Gabriel Mazetto 1505720787 +0200
    committer Gabriel Mazetto 1505721333 +0200
    Added GitAdapter, fixed things and now ls-remote works.

    View Slide

  25. 74ac5c4
    commit
    f7530ab
    tree
    ffd2a4a
    blob
    1cdeb2b
    blob
    2d3df47
    commit
    615b472
    tree eb5bce0
    blob
    ...

    View Slide

  26. How is the hash
    generated?

    View Slide

  27. SHA1([obj-type][size][NUL][object-content])

    View Slide

  28. $ printf "commit %s\0" $(git cat-file
    commit HEAD | wc -c)
    commit 279
    NUL byte character

    View Slide

  29. $ (printf "commit %s\0" $(git cat-file commit HEAD | wc -c);
    git cat-file commit HEAD)
    commit 279tree f7530ab4de2105b49789dda745fb450b95c2a33b
    parent 2d3df47cef64497c77f52f0803e2ca1467f5be1a
    author Gabriel Mazetto 1505724591 +0200
    committer Gabriel Mazetto 1505724591 +0200
    Added support for Git RPC with upload-pack (clone works now)

    View Slide

  30. $ (printf "commit %s\0" $(git cat-file commit HEAD | wc -c);
    git cat-file commit HEAD) | sha1sum
    74ac5c498323b39dfb14eec9244b5f628b717555
    $ git show-ref --head --hash HEAD
    74ac5c498323b39dfb14eec9244b5f628b717555

    View Slide

  31. View Slide

  32. Let's talk about protocols...

    View Slide

  33. Syncing your

    View Slide

  34. git, ssh, https*
    WebDAV, dumb, smart

    View Slide

  35. The git:// protocol
    $ git daemon --base-path=/srv/git
    READONLY

    View Slide

  36. The ssh:// protocol
    $ git-upload-pack
    $ git-receive-pack

    View Slide

  37. The webdav:// protocol

    View Slide

  38. The dumb protocol
    $ update-server-info
    READONLY

    View Slide

  39. The smart protocol
    $ git-upload-pack
    $ git-receive-pack

    View Slide

  40. Let's talk about
    C

    View Slide

  41. Single Binary
    $ caddy -conf /path/to/Caddyfile

    View Slide

  42. It's not only a
    HTTP

    View Slide

  43. Secure by Default!
    L ' E
    MITM attack detection

    View Slide

  44. label1 {
    directive1 arg1
    directive2 arg1 arg2
    directive3 {
    subdir1 arg1 arg2
    subdir2 arg1 arg2
    }
    }

    View Slide

  45. https://awesomeapp.example.com {
    root /srv/awesomeapp.example.com
    log /var/opt/log/awesomeapp.log {
    rotate_size 50 # after 50MB
    rotate_keep 20 # log files to keep
    rotate_compress # gzip after rotation
    }
    proxy / localhost:3000 localhost:3001 {
    policy round_robin # rotate requests
    transparent # forward request IPs
    }
    }

    View Slide

  46. Now, try to do the same
    config with nginx...

    View Slide

  47. View Slide

  48. Let's try to build some
    f ...

    View Slide

  49. There are few types of
    plugins

    View Slide

  50. We are going to extend the
    Caddyfile and add a HTTP
    Middleware

    View Slide

  51. The Boilerplate
    https://gitlab.com/brodock/caddy-helloworld

    View Slide

  52. The SMART HTTP Proxy
    https://gitlab.com/brodock/caddy-gitproxy

    View Slide

  53. Q ?

    View Slide