Save 37% off PRO during our Black Friday Sale! »

Dokku Internals

Dokku Internals

A technical talk where I'll be going over what happens between when a user performs a "git push" and the app becomes available for viewing. Topics will include the Heroku buildpack lifecycle, docker container usage, and the Dokku plugin system.


Jose Diaz-Gonzalez

July 30, 2019


  1. Dokku Internals Random acts of shell scripts

  2. What is Dokku Exactly? Or what isn’t it

  3. What is Dokku? • A set of executables acting as

    a pipeline • Dokku takes some input and produces one or more artifacts • A large collection of shell scripts
  4. What isn’t Dokku • Not a Platform as a Service,

    but commonly used as one • A promoter of whale sushi • Strictly docker-based technology
  5. Some History Because reading is fundamental

  6. History • First “killer app” for docker • Initially strung

    together Git + Nginx + Docker + Buildpacks with ~100 lines of bash • Supports deploys to Docker, Kubernetes, Nomad • 5k commits, 17k+ stars, 140 releases, 200+ community plugins
  7. History • 2013 : Started by Jeff Lindsay (@progrium) •

    2014: Taken over by Jose Gonzalez • 2015: Official Datastore plugins • 2016: Zero downtime deploys • 2017: Started slow rewrite into golang • 2018: Herokuish 0.4.0/Ubuntu 16.04 base • 2019: Kubernetes/Nomad support
  8. Dokku Workflow Mechanism How does it know what to do?

  9. git push dokku master • On git push, associated ssh

    key triggers a command configured in /home/ dokku/.sshcommand • dokku binary implements git-receive-pack and writes code to /home/dokku/$APP • /home/dokku/$APP/hooks/pre-receive contains code that triggers the build process, and streams back output to stdout • dokku/git-http-backend (OSS) provides http(s) support, writing the code to the $APP directory, and chunks back output over http(s) to the client • This bypasses ssh key check, authentication must be performed via basic-auth or similar within a proxy in front of the git-http-backend. • Only master branch is built, but this can be configured • dokku git:set $APP deploy-branch $NEW_BRANCH
  10. Dokku command overview • Dokku commands provide the majority of

    the Heroku-like experience. They wrap common infrastructure tasks that are hard to remember/easy to mix up. • This is an alternative to using tools such as Ansible, Chef, or Puppet, though may be used in conjunction with them (e.g. ansible-dokku) • dokku binary takes many arguments. First argument is the subcommand, all others are passed to subcommand. Some simple flag parsing is also performed here. • Subcommands can be any executable code, but commonly shell. • Subcommands are grouped by common functionality • The git plugin contains triggers and subcommands related to git, while the config plugin handles environment variable management.
  11. Extending Subcommands • Subcommands can call any “exported” functions from

    other plugins • Convention is to place exportable functions in a functions file, while non-exportable functions should be in internal-functions • Subcommands can also “call” plugin triggers via plugn binary to avoid sourcing other plugins • Plugin triggers are commonly used to enable/disable functionality
  12. Plugin Internals • All plugin triggers are captured as events

    in a log via the events plugin (dokku events:on) • Core plugins can be swapped out for alternative implementations (dokku-scheduler-kubernetes) • Core plugins generally are not disabled, but extended via triggers • Alternative implementations may result in a user disabling a core plugin. A good example would be an alternative proxy implementation around caddy or haproxy • Core plugins provide helper functions for community plugins to utilize (config/property management)
  13. Plugin Examples • Smoke Test Plugin: smoke-test-plugin • Postgres

    Datastore: dokku-postgres • Kubernetes Scheduler: dokku-scheduler-kubernetes • See plugin-creation/ for plugin creation tips
  14. Simplest plugin post-deploy bash script

  15. Buildpacks The secret sauce to app building

  16. History • Started by Heroku in 2010/2011 for their PaaS

    • Two standards now • Heroku: Buildpack v2a • Cloud Foundry: Buildpack v2b • New joint standard (v3) under development • See for more information
  17. Buildpack v2A • A bundle of executables • bin/detect: detects

    whether the buildpack applies • bin/compile: compiles dependencies and app code • bin/release: specifies default processes and platform addons for the app • Generally also includes supporting building code • Normally only targets a single language
  18. Who maintains buildpacks? • Heroku has a collection of 10

    official buildpacks, any other unofficial ones • CloudFoundry maintains their own v2b buildpacks • Hundreds of Community Buildpacks • Dokku project maintains the static buildpack • Plotly maintains custom Conda, Python, and R buildpacks • Maybe you?
  19. Buildpack distribution • Generally distributed as git repositories • Buildpacks

    are installed on “stacks”, or base operating systems • gliderlabs/herokuish provides a docker- compatible layer for buildpacks (Dokku uses this) • Buildpack v3 will have officially provided stacks with built-in buildpacks
  20. Buildpack stacks • Provides ABI compatibility for compiled dependencies •

    Provides base OS + common utilities necessary to run an app • v2a Current options: cedar-10, cedar-14, heroku-16, heroku-18 • v3 options: bionic stack (heroku maintained)
  21. Herokuish • Herokuish is: • A collection of buildpacks •

    A binary that can act as an init for apps • A docker image • Released in a single stack flavor (currently heroku-18) • Can be further customized for dokku via DOKKU_IMAGE environment variable • Will be deprecated in favor of buildpack v3 pack-cli or equivalent
  22. Buildpack output • Binaries or source + dependencies • Golang/Java:

    Binary • R/Python: Source + dependencies • Export Formats • Cloud Foundry: Exports a tarball “droplet” of /app directory • Heroku: Exports a tarball “slug" of /app directory • Dokku: Provides a docker image of all changes • Dokku is the odd one here
  23. Docker In Dokku “What is it good for? Absolutely nothing!”

  24. Image Building Buildpacks avoid Dockerfiles, so how do we create

    a docker image?
  25. Intermediate Containers • docker run/wait/commit • Run a command inside

    of a container and commits output to docker image • Workflow allows for continually adding/removing from output image (e.g. dokku-apt) • Built image no different from any other docker image
  26. Scheduling • Default to docker-local scheduling • docker run --restart-policy

    $IMAGE… • Kubernetes support via plugin • Depends on the (external but official) registry plugin and an external registry • Supports both process and resource scaling • Currently requires a custom routing implementation • Nomad support via plugin • Similar requirements as Kubernetes support • Others?
  27. Datastores? • Datastores are separate containers running with mounted directories

    for storage • All common datastores are currently implemented as OSS plugins (Elasticsearch, MySQL, Postgres, Redis, etc.) • Upcoming Couchbase, Neo4j, Omnisci • Old docker --link flag used for communication • Sometimes does not withstand datastore restarts • Upcoming release will switch to non-deprecated networks • No clustering support on OSS datastores plugins • No resource limits specific on OSS datastores plugins • Should there be?
  28. Cross-container communication • Not possible except with: • intermediary (queuing

    through redis) • plain old http over open network • Commonly requested OSS feature (two apps talking to each other) • Maybe not as requested in commercial settings? • Upcoming release will implement official networking support • Strictly opt-in, but not multi-tenant
  29. Cross-server communication • If the server has access to it,

    so should the apps • No special docker0 rules in place to block inbound/outbound • This applies to IAM roles and other things living on the server • Outgoing access can be controlled via iptables • Many security softwares can break docker0, causing invalid networking for docker containers! • Other access control should be done via password or firewall
  30. thats it