Building a git reverse proxy in Go for Caddy

Scaling non-trivial software can be complex. When your core business depends on custom communication protocols, you may end up having to get your hands dirty and write custom system services. In this talk I’m going to show how we can leverage the modular Caddy HTTP server and implement a Git reverse proxy bundled in it. You may also learn a few things about Git internals you never had the chance to ask anyone about.

Links at the end of the presentation:
The Boiler Plate: https://gitlab.com/brodock/caddy-helloworld
The SMART HTTP Proxy: https://gitlab.com/brodock/caddy-gitproxy

Transcript

2. Building a git reverse proxy G f C
3. @brodock blog.gabrielmazetto.eti.br Gabriel Mazetto
5. That version control thing...
6. This is a talk about git and its internals f G ...
7. I WILL NOT TEACH YOU HOW TO
9. I have to introduce you to a lot of concepts before...
10. $ git help clone Clone a repository into a new directory init Create an empty Git repository or reinitialize an existing one add Add file contents to the index mv Move or rename a file, a directory, or a symlink reset Reset current HEAD to the specified state rm Remove files from the working tree and from the index log Show commit logs show Show various types of objects status Show the working tree status branch List, create, or delete branches checkout Switch branches or restore working tree files commit Record changes to the repository diff Show changes between commits, commit and working tree, etc merge Join two or more development histories together rebase Forward-port local commits to the updated upstream head tag Create, list, delete or verify a tag object signed with GPG fetch Download objects and refs from another repository pull Fetch from and integrate with another repository or a local branch push Update remote refs along with associated objects
11. What git really is?
12. $ find bin/ libexec/ | wc -l 76 not a single binary file...
14. Unix Philosophy: S , ,
15. $ find bin/ libexec/ ./bin/git ./bin/git-cvsserver ./bin/git-receive-pack ./bin/git-shell ./bin/git-upload-archive ./bin/git-upload-pack ./libexec/gitcore/git-rebase ./libexec/gitcore/git-stash ./libexec/gitcore/git-submodule ./libexec/gitcore/git-mergetool ./libexec/gitcore/git-bisect ./libexec/gitcore/git-daemon ./libexec/gitcore/git-rebase--am ./libexec/gitcore/git-parse-remote ./libexec/gitcore/git-svn ./libexec/gitcore/git-http-backendu ... perl shell script
16. Where is my code ?
17. P Object Database (ODB)
18. How current ODB backend work?
19. $ git log commit 74ac5c498323b39dfb14eec9244b5f628b717555 Author: Gabriel Mazetto <brodock@gmail.com> Date: Mon Sep 18 10:49:51 2017 +0200 Added support for Git RPC with upload-pack (clone works now) type SHA1 hash (key) content (data)
20. So is this how the commit object look like?
21. $ git show-ref --head --hash HEAD 74ac5c498323b39dfb14eec9244b5f628b717555 $ git cat-file commit HEAD tree f7530ab4de2105b49789dda745fb450b95c2a33b parent 2d3df47cef64497c77f52f0803e2ca1467f5be1a author Gabriel Mazetto <brodock@gmail.com> 1505724591 +0200 committer Gabriel Mazetto <brodock@gmail.com> 1505724591 +0200 Added support for Git RPC with upload-pack (clone works now)
22. $ git cat-file -p f7530ab4de2105b49789dda745fb450b95c2a33b 100644 blob ffd2a4a2e1b6f84ec571a84a4bd6f001643a0745 .gitignore 100644 blob 0929ab8179725c680176ea5001dbab28faea166d README.md 100644 blob 1cdeb2b8f92771a60fe7152f541fdb939f185b86 git.go 100644 blob 7ddbc56e43e5b6ab8e7e00c88589f78c2b7eef04 gitproxy.go 100644 blob 76133a6764e0e49b787dbb49a3103f64f83246f1 setup.go 100644 blob 18b3b795d9a196ecfaf9548be5f721f3f878d9ae setup_test.go
23. $ git show-ref --head --hash HEAD 74ac5c498323b39dfb14eec9244b5f628b717555 $ git cat-file commit HEAD tree f7530ab4de2105b49789dda745fb450b95c2a33b parent 2d3df47cef64497c77f52f0803e2ca1467f5be1a author Gabriel Mazetto <brodock@gmail.com> 1505724591 +0200 committer Gabriel Mazetto <brodock@gmail.com> 1505724591 +0200 Added support for Git RPC with upload-pack (clone works now)
24. $ git cat-file 2d3df47cef64497c77f52f0803e2ca1467f5be1a tree 615b472d533b66b908f7d79a0b9d18be32779de4 parent 7fa2356cc5ecd80e949803447dfeb8b566764186 author Gabriel Mazetto <brodock@gmail.com> 1505720787 +0200 committer Gabriel Mazetto <brodock@gmail.com> 1505721333 +0200 Added GitAdapter, fixed things and now ls-remote works.
25. 74ac5c4 commit f7530ab tree ffd2a4a blob 1cdeb2b blob 2d3df47 commit 615b472 tree eb5bce0 blob ...
26. How is the hash generated?
27. SHA1([obj-type][size][NUL][object-content])
28. $ printf "commit %s\0" $(git cat-file commit HEAD | wc -c) commit 279 NUL byte character
29. $ (printf "commit %s\0" $(git cat-file commit HEAD | wc -c); git cat-file commit HEAD) commit 279tree f7530ab4de2105b49789dda745fb450b95c2a33b parent 2d3df47cef64497c77f52f0803e2ca1467f5be1a author Gabriel Mazetto <brodock@gmail.com> 1505724591 +0200 committer Gabriel Mazetto <brodock@gmail.com> 1505724591 +0200 Added support for Git RPC with upload-pack (clone works now)
30. $ (printf "commit %s\0" $(git cat-file commit HEAD | wc -c); git cat-file commit HEAD) | sha1sum 74ac5c498323b39dfb14eec9244b5f628b717555 $ git show-ref --head --hash HEAD 74ac5c498323b39dfb14eec9244b5f628b717555
32. Let's talk about protocols...
33. Syncing your
34. git, ssh, https* WebDAV, dumb, smart
35. The git:// protocol $ git daemon --base-path=/srv/git READONLY
36. The ssh:// protocol $ git-upload-pack $ git-receive-pack
37. The webdav:// protocol
38. The dumb protocol $ update-server-info READONLY
39. The smart protocol $ git-upload-pack $ git-receive-pack
40. Let's talk about C
41. Single Binary $ caddy -conf /path/to/Caddyfile
42. It's not only a HTTP
43. Secure by Default! L ' E MITM attack detection
44. label1 { directive1 arg1 directive2 arg1 arg2 directive3 { subdir1 arg1 arg2 subdir2 arg1 arg2 } }
45. https://awesomeapp.example.com { root /srv/awesomeapp.example.com log /var/opt/log/awesomeapp.log { rotate_size 50 # after 50MB rotate_keep 20 # log files to keep rotate_compress # gzip after rotation } proxy / localhost:3000 localhost:3001 { policy round_robin # rotate requests transparent # forward request IPs } }
46. Now, try to do the same config with nginx...
48. Let's try to build some f ...
49. There are few types of plugins
50. We are going to extend the Caddyfile and add a HTTP Middleware
51. The Boilerplate https://gitlab.com/brodock/caddy-helloworld
52. The SMART HTTP Proxy https://gitlab.com/brodock/caddy-gitproxy
53. Q ?