Docker Lessons from Real-World Projects

Docker Lessons from Real-World Projects

Presented at DjangoCon Europe 2017

Explores how to use Docker for complex, real-world Django applications.

895ac7b84a280f686b70e2a70a8f5cd9?s=128

Rivo Laks

April 05, 2017
Tweet

Transcript

  1. Docker Lessons from Real-World Projects Rivo Laks ⋅ Thorgate 2017-04-05

  2. The Problem

  3. The Problem At Thorgate, we host dozens of our projects

  4. The Problem At Thorgate, we host dozens of our projects

    A single server runs multiple projects
  5. The Problem At Thorgate, we host dozens of our projects

    A single server runs multiple projects Dependency isolation became an issue
  6. So we thought – Docker!?

  7. Why Docker? provides isolation

  8. Why Docker? provides isolation exotic services easier to use

  9. Why Docker? provides isolation exotic services easier to use project

    bundled into a single image
  10. How to Get There? Lots of online tutorials for making

    rst steps Practical experience with bigger projects is much more scarce
  11. Wait, What's Docker?

  12. None
  13. None
  14. None
  15. Your App on Docker

  16. Application Layout Few assumptions:

  17. Application Layout Few assumptions: Django, Node.js, database, other services

  18. Application Layout Few assumptions: Django, Node.js, database, other services Single-server

    apps only
  19. Application Layout Few assumptions: Django, Node.js, database, other services Single-server

    apps only Looking at production rst
  20. None
  21. The Django Service

  22. The Django Service Start with o cial python image

  23. The Django Service Start with o cial python image Install

    Python requirements
  24. The Django Service Start with o cial python image Install

    Python requirements Add application source code
  25. The Django Service Start with o cial python image Install

    Python requirements Add application source code Use Gunicorn to serve HTTP
  26. The Django Service Start with o cial python image Install

    Python requirements Add application source code Use Gunicorn to serve HTTP Volumes are used for data dirs This includes static assets, media and logs
  27. The Node Service Bundles JS + CSS using Webpack

  28. The Node Service Bundles JS + CSS using Webpack Isn't

    run continuously, just once during deploy
  29. The Node Service Bundles JS + CSS using Webpack Isn't

    run continuously, just once during deploy Basically o cial Node image + our requirements
  30. The Node Service Bundles JS + CSS using Webpack Isn't

    run continuously, just once during deploy Basically o cial Node image + our requirements Volumes for JS/CSS source dirs
  31. Other Services Redis, Celery, etc

  32. Other Services Redis, Celery, etc Can often use o cial

    image out-of-the-box
  33. Other Services Redis, Celery, etc Can often use o cial

    image out-of-the-box Volumes for persistent data
  34. Putting it All Together Our application has multiple services working

    in unison, communicating via network
  35. Putting it All Together Our application has multiple services working

    in unison, communicating via network Docker Compose is “a tool for de ning and running multi-container Docker applications”
  36. Putting it All Together Our application has multiple services working

    in unison, communicating via network Docker Compose is “a tool for de ning and running multi-container Docker applications” de ne & control multiple containers at once
  37. Putting it All Together Our application has multiple services working

    in unison, communicating via network Docker Compose is “a tool for de ning and running multi-container Docker applications” de ne & control multiple containers at once easier networking among single app's services
  38. None
  39. Docker Compose – Examples docker-compose build

  40. Docker Compose – Examples docker-compose build docker-compose up

  41. Docker Compose – Examples docker-compose build docker-compose up docker-compose logs

  42. Docker Compose – Examples docker-compose build docker-compose up docker-compose logs

    docker-compose run django ./manage.py migrate
  43. The Servers Almost nothing but Docker

  44. The Servers Almost nothing but Docker Postgres and Nginx as

    well, since we chose to leave those global
  45. Deploys

  46. Deploys docker-compose build

  47. Deploys docker-compose build docker-compose run --rm node npm run build

  48. Deploys docker-compose build docker-compose run --rm node npm run build

    docker-compose run --rm django ./manage.py collectstatic docker-compose run --rm django ./manage.py migrate
  49. Deploys docker-compose build docker-compose run --rm node npm run build

    docker-compose run --rm django ./manage.py collectstatic docker-compose run --rm django ./manage.py migrate docker-compose up -d
  50. Development Environment

  51. Dev vs Production Should ideally be very similar, but some

    di erences are useful
  52. Dev vs Production Should ideally be very similar, but some

    di erences are useful Commands are slightly di erent
  53. Dev vs Production Should ideally be very similar, but some

    di erences are useful Commands are slightly di erent Sources not baked into images
  54. Dev vs Production Should ideally be very similar, but some

    di erences are useful Commands are slightly di erent Sources not baked into images Postgres is run locally, along other services
  55. Dev vs Production Should ideally be very similar, but some

    di erences are useful Commands are slightly di erent Sources not baked into images Postgres is run locally, along other services Separate Docker Compose con g le
  56. Django App dir is mounted as volume, not baked into

    image
  57. Django App dir is mounted as volume, not baked into

    image Python dependencies still part of the image
  58. Django App dir is mounted as volume, not baked into

    image Python dependencies still part of the image Django's runserver instead of Gunicorn
  59. Django App dir is mounted as volume, not baked into

    image Python dependencies still part of the image Django's runserver instead of Gunicorn Separate Dockerfile
  60. Node Image contains libraries but not our sources (same as

    production)
  61. Node Image contains libraries but not our sources (same as

    production) Webpack watch mode, similar to Django's runserver
  62. Other Services Similar to production Postgres is run locally

  63. Other Services Similar to production Postgres is run locally Project-speci

    c volumes for data
  64. Quirks aka The Death by a Thousand Papercuts

  65. Commands docker-compose run --rm django ./manage.py runserver docker-compose run --rm

    django ./manage.py migrate
  66. Commands docker-compose run --rm django ./manage.py runserver docker-compose run --rm

    django ./manage.py migrate Make le FTW! make docker make migrate
  67. Commands docker-compose run --rm django ./manage.py runserver docker-compose run --rm

    django ./manage.py migrate Make le FTW! make docker make migrate Bonus: make setup
  68. Service Discovery How to route HTTP requests to correct Django

    container
  69. Service Discovery How to route HTTP requests to correct Django

    container Ran Nginx in container, used Docker's built-in routing
  70. Service Discovery How to route HTTP requests to correct Django

    container Ran Nginx in container, used Docker's built-in routing But container IP can change, Nginx does DNS lookup only once
  71. Service Discovery How to route HTTP requests to correct Django

    container Ran Nginx in container, used Docker's built-in routing But container IP can change, Nginx does DNS lookup only once Created a service monitoring Docker events and reloading Nginx
  72. Waiting for Related Services Need to wait for Postgres to

    be available for Django
  73. Waiting for Related Services Need to wait for Postgres to

    be available for Django wait-for-it.sh scripts
  74. Performance on non-Linux OSs On macOS / Windows, Docker containers

    run in VM
  75. Performance on non-Linux OSs On macOS / Windows, Docker containers

    run in VM Filesystem performance su ers
  76. Performance on non-Linux OSs On macOS / Windows, Docker containers

    run in VM Filesystem performance su ers localhost isn't localhost anymore
  77. PyCharm Supports Python interpreter in Docker

  78. PyCharm Supports Python interpreter in Docker There are bugs though

  79. PyCharm Supports Python interpreter in Docker There are bugs though

    Running app inside PyCharm was tricky
  80. PyCharm Supports Python interpreter in Docker There are bugs though

    Running app inside PyCharm was tricky node_modules isn't visible to PyCharm
  81. More Tooling Plus 666 Other Issues...

  82. Should I Use Docker?

  83. Problems Learning curve – you need to learn Docker

  84. Problems Learning curve – you need to learn Docker You'll

    also spend time on interesting new issues
  85. Problems Learning curve – you need to learn Docker You'll

    also spend time on interesting new issues Ongoing problems: Tooling
  86. Problems Learning curve – you need to learn Docker You'll

    also spend time on interesting new issues Ongoing problems: Tooling Performance outside Linux
  87. Advantages Isolation & reproducability

  88. Advantages Isolation & reproducability Easier to add new services

  89. Advantages Isolation & reproducability Easier to add new services Bundling

    everything into a single entity
  90. Advantages Isolation & reproducability Easier to add new services Bundling

    everything into a single entity Limiting/monitoring resource use is easier
  91. Advantages Isolation & reproducability Easier to add new services Bundling

    everything into a single entity Limiting/monitoring resource use is easier Single-command project setup
  92. It Depends...

  93. It Depends... For legacy projects, maybe not

  94. It Depends... For legacy projects, maybe not But makes more

    sense for new projects
  95. It Depends... For legacy projects, maybe not But makes more

    sense for new projects Thorgate open-sourcing our project template
  96. Can You Spare a New Tech Point?

  97. Thanks! Rivo Laks ⋅ @rivolaks ⋅ rivolaks.com CTO at Thorgate

    ⋅ thorgate.eu https://github.com/thorgate/django-project-template