Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Dockercon SF 2018: Self-upgrading Servers with Docker

Dockercon SF 2018: Self-upgrading Servers with Docker

Discover how we implemented a server that doesn’t require any downtime to be extended with new features.

Babelfish (bblf.sh) is an open source project created at source{d} that provides language parsing for *any* programming language.

Each parser is written in its own language, so we can’t just simply put everything in a binary.

So how does this work?

We are able to update and upgrade the API server by running Moby inside of Moby (we need to go deeper!) and using the Docker hub as our repository for parsers.

In addition to this, each container image for parsers needs to follow some quite strict specifications: and in order to do so we build a set of images via Makefile and multi-stage builds.

All in all creates an efficient system that although is architecturally complex provides a simple developer experience.

This talk covers all the juicy details of the architecture and the lessons we learned while developing it.

D8e5d79ca42edc07693b9c1aacaa7e5e?s=128

Francesc Campoy Flores

June 14, 2018
Tweet

Transcript

  1. Hello, I’m Francesc

  2. Machine Learning on Source Code #MLonCode

  3. '112', '97', '99', '107', '97', '103', '101', '32', '109', '97',

    '105', '110', '10', '10', '105', '109', '112', '111', '114', '116', '32', '40', '10', '9', '34', '102', '109', '116', '34', '10', '41', '10', '10', '102', '117', '110', '99', '32', '109', '97', '105', '110', '40', '41', '32', '123', '10', '9', '102', '109', '116', '46', '80', '114', '105', '110', '116', '108', '110', '40', '34', '72', '101', '108', '108', '111', '44', '32', '112', '108', '97', '121', '103', '114', '111', '117', '110', '100', '34', '41', '10', '125', '10' package main import “fmt” func main() { fmt.Println(“Hello, Chicago”) } What is Source Code
  4. package package IDENT main ; import import STRING "fmt" ;

    func func IDENT main ( ) package main import “fmt” func main() { fmt.Println(“Hello, Chicago”) } What is Source Code { IDENT fmt . IDENT Println ( STRING "Hello, Chicago" ) ; } ;
  5. package main import “fmt” func main() { fmt.Println(“Hello, Chicago”) }

    What is Source Code
  6. None
  7. … a self-hosted server for universal source code parsing, turning

    code files into Universal Abstract Syntax Trees
  8. None
  9. None
  10. None
  11. Parsing Go in Go is easy Parsing Java in Java

    is easy Parsing Go in Java is doable … I guess Parsing Java in Go is interesting … Parsing C++ in Go is why are you like this? Parsing X in Y
  12. N parsers in 1 language is way harder than N

    parsers in M languages
  13. None
  14. None
  15. • Good UX ◦ Intuitive commands, flags, args ... ◦

    Reasonable defaults • Frictionless installation ◦ As few dependencies as possible ◦ One line to run is good ◦ No lines to run is even better DevRel: reducing time to “wow”!
  16. docker run >>> apt-get + pip + brew + ....

  17. Bblfsh container docker run foo container foo

  18. None
  19. Docker in Docker! Using Docker-in-Docker for your CI or testing

    environment? Think twice jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/
  20. Docker in Docker! However, contrary to popular belief, Docker-in-Docker is

    not 100% made of sparkles, ponies, and unicorns
  21. Actually … docker run \ --privileged -d \ docker:stable-dind Later

    on ... hub.docker.com/r/library/docker/
  22. Docker in Docker! jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/ Just map /var/run/docker.sock !

  23. container Docker CLI /var/run/docker.sock

  24. container docker run foo /var/run/docker.sock container foo

  25. libcontainer

  26. None
  27. None
  28. Managing drivers (parsers)

  29. Scaling • Pluggable strategy • Default behavior: ◦ 1 container

    per request ◦ Extra time for reuse • Not implemented yet • Upgrade is for now Remove + Installation bblfshctl driver \ install --update \ foo Upgrading
  30. Docker in Docker Docker Socket - Hard to use, previously

    - Much easier now - Lots of features we don’t use - Requires Docker to be used no matter what libcontainer - Fewer features - Fewer dependencies - Runs on Linux without Docker
  31. References - bblf.sh / dashboard.bblf.sh - sourced.tech - github.com/bblfsh -

    @francesc @sourcedtech
  32. Thanks!