Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Docker Lessons from Real-World Projects

Docker Lessons from Real-World Projects

Presented at DjangoCon Europe 2017

Explores how to use Docker for complex, real-world Django applications.

Rivo Laks

April 05, 2017
Tweet

More Decks by Rivo Laks

Other Decks in Technology

Transcript

  1. Docker Lessons
    from Real-World Projects
    Rivo Laks ⋅ Thorgate
    2017-04-05

    View Slide

  2. The Problem

    View Slide

  3. The Problem
    At Thorgate, we host dozens of our projects

    View Slide

  4. The Problem
    At Thorgate, we host dozens of our projects
    A single server runs multiple projects

    View Slide

  5. The Problem
    At Thorgate, we host dozens of our projects
    A single server runs multiple projects
    Dependency isolation became an issue

    View Slide

  6. So we thought – Docker!?

    View Slide

  7. Why Docker?
    provides isolation

    View Slide

  8. Why Docker?
    provides isolation
    exotic services easier to use

    View Slide

  9. Why Docker?
    provides isolation
    exotic services easier to use
    project bundled into a single image

    View Slide

  10. How to Get There?
    Lots of online tutorials for making rst steps
    Practical experience with bigger projects is much more
    scarce

    View Slide

  11. Wait, What's Docker?

    View Slide

  12. View Slide

  13. View Slide

  14. View Slide

  15. Your App on Docker

    View Slide

  16. Application Layout
    Few assumptions:

    View Slide

  17. Application Layout
    Few assumptions:
    Django, Node.js, database, other services

    View Slide

  18. Application Layout
    Few assumptions:
    Django, Node.js, database, other services
    Single-server apps only

    View Slide

  19. Application Layout
    Few assumptions:
    Django, Node.js, database, other services
    Single-server apps only
    Looking at production rst

    View Slide

  20. View Slide

  21. The Django Service

    View Slide

  22. The Django Service
    Start with o cial python image

    View Slide

  23. The Django Service
    Start with o cial python image
    Install Python requirements

    View Slide

  24. The Django Service
    Start with o cial python image
    Install Python requirements
    Add application source code

    View Slide

  25. The Django Service
    Start with o cial python image
    Install Python requirements
    Add application source code
    Use Gunicorn to serve HTTP

    View Slide

  26. The Django Service
    Start with o cial python image
    Install Python requirements
    Add application source code
    Use Gunicorn to serve HTTP
    Volumes are used for data dirs
    This includes static assets, media and logs

    View Slide

  27. The Node Service
    Bundles JS + CSS using Webpack

    View Slide

  28. The Node Service
    Bundles JS + CSS using Webpack
    Isn't run continuously, just once during deploy

    View Slide

  29. The Node Service
    Bundles JS + CSS using Webpack
    Isn't run continuously, just once during deploy
    Basically o cial Node image + our requirements

    View Slide

  30. The Node Service
    Bundles JS + CSS using Webpack
    Isn't run continuously, just once during deploy
    Basically o cial Node image + our requirements
    Volumes for JS/CSS source dirs

    View Slide

  31. Other Services
    Redis, Celery, etc

    View Slide

  32. Other Services
    Redis, Celery, etc
    Can often use o cial image out-of-the-box

    View Slide

  33. Other Services
    Redis, Celery, etc
    Can often use o cial image out-of-the-box
    Volumes for persistent data

    View Slide

  34. Putting it All Together
    Our application has multiple services working in
    unison, communicating via network

    View Slide

  35. Putting it All Together
    Our application has multiple services working in
    unison, communicating via network
    Docker Compose is “a tool for de ning and running
    multi-container Docker applications”

    View Slide

  36. Putting it All Together
    Our application has multiple services working in
    unison, communicating via network
    Docker Compose is “a tool for de ning and running
    multi-container Docker applications”
    de ne & control multiple containers at once

    View Slide

  37. Putting it All Together
    Our application has multiple services working in
    unison, communicating via network
    Docker Compose is “a tool for de ning and running
    multi-container Docker applications”
    de ne & control multiple containers at once
    easier networking among single app's services

    View Slide

  38. View Slide

  39. Docker Compose – Examples
    docker-compose build

    View Slide

  40. Docker Compose – Examples
    docker-compose build
    docker-compose up

    View Slide

  41. Docker Compose – Examples
    docker-compose build
    docker-compose up
    docker-compose logs

    View Slide

  42. Docker Compose – Examples
    docker-compose build
    docker-compose up
    docker-compose logs
    docker-compose run django ./manage.py migrate

    View Slide

  43. The Servers
    Almost nothing but Docker

    View Slide

  44. The Servers
    Almost nothing but Docker
    Postgres and Nginx as well, since we chose to leave
    those global

    View Slide

  45. Deploys

    View Slide

  46. Deploys
    docker-compose build

    View Slide

  47. Deploys
    docker-compose build
    docker-compose run --rm node npm run build

    View Slide

  48. Deploys
    docker-compose build
    docker-compose run --rm node npm run build
    docker-compose run --rm django ./manage.py collectstatic
    docker-compose run --rm django ./manage.py migrate

    View Slide

  49. Deploys
    docker-compose build
    docker-compose run --rm node npm run build
    docker-compose run --rm django ./manage.py collectstatic
    docker-compose run --rm django ./manage.py migrate
    docker-compose up -d

    View Slide

  50. Development
    Environment

    View Slide

  51. Dev vs Production
    Should ideally be very similar, but some di erences
    are useful

    View Slide

  52. Dev vs Production
    Should ideally be very similar, but some di erences
    are useful
    Commands are slightly di erent

    View Slide

  53. Dev vs Production
    Should ideally be very similar, but some di erences
    are useful
    Commands are slightly di erent
    Sources not baked into images

    View Slide

  54. Dev vs Production
    Should ideally be very similar, but some di erences
    are useful
    Commands are slightly di erent
    Sources not baked into images
    Postgres is run locally, along other services

    View Slide

  55. Dev vs Production
    Should ideally be very similar, but some di erences
    are useful
    Commands are slightly di erent
    Sources not baked into images
    Postgres is run locally, along other services
    Separate Docker Compose con g le

    View Slide

  56. Django
    App dir is mounted as volume, not baked into image

    View Slide

  57. Django
    App dir is mounted as volume, not baked into image
    Python dependencies still part of the image

    View Slide

  58. Django
    App dir is mounted as volume, not baked into image
    Python dependencies still part of the image
    Django's runserver instead of Gunicorn

    View Slide

  59. Django
    App dir is mounted as volume, not baked into image
    Python dependencies still part of the image
    Django's runserver instead of Gunicorn
    Separate Dockerfile

    View Slide

  60. Node
    Image contains libraries but not our sources
    (same as production)

    View Slide

  61. Node
    Image contains libraries but not our sources
    (same as production)
    Webpack watch mode, similar to Django's runserver

    View Slide

  62. Other Services
    Similar to production
    Postgres is run locally

    View Slide

  63. Other Services
    Similar to production
    Postgres is run locally
    Project-speci c volumes for data

    View Slide

  64. Quirks
    aka The Death by a Thousand Papercuts

    View Slide

  65. Commands
    docker-compose run --rm django ./manage.py runserver
    docker-compose run --rm django ./manage.py migrate

    View Slide

  66. Commands
    docker-compose run --rm django ./manage.py runserver
    docker-compose run --rm django ./manage.py migrate
    Make le FTW!
    make docker
    make migrate

    View Slide

  67. Commands
    docker-compose run --rm django ./manage.py runserver
    docker-compose run --rm django ./manage.py migrate
    Make le FTW!
    make docker
    make migrate
    Bonus: make setup

    View Slide

  68. Service Discovery
    How to route HTTP requests to correct Django container

    View Slide

  69. Service Discovery
    How to route HTTP requests to correct Django container
    Ran Nginx in container, used Docker's built-in routing

    View Slide

  70. Service Discovery
    How to route HTTP requests to correct Django container
    Ran Nginx in container, used Docker's built-in routing
    But container IP can change, Nginx does DNS lookup
    only once

    View Slide

  71. Service Discovery
    How to route HTTP requests to correct Django container
    Ran Nginx in container, used Docker's built-in routing
    But container IP can change, Nginx does DNS lookup
    only once
    Created a service monitoring Docker events and
    reloading Nginx

    View Slide

  72. Waiting for Related Services
    Need to wait for Postgres to be available for Django

    View Slide

  73. Waiting for Related Services
    Need to wait for Postgres to be available for Django
    wait-for-it.sh scripts

    View Slide

  74. Performance on non-Linux OSs
    On macOS / Windows, Docker containers run in VM

    View Slide

  75. Performance on non-Linux OSs
    On macOS / Windows, Docker containers run in VM
    Filesystem performance su ers

    View Slide

  76. Performance on non-Linux OSs
    On macOS / Windows, Docker containers run in VM
    Filesystem performance su ers
    localhost isn't localhost anymore

    View Slide

  77. PyCharm
    Supports Python interpreter in Docker

    View Slide

  78. PyCharm
    Supports Python interpreter in Docker
    There are bugs though

    View Slide

  79. PyCharm
    Supports Python interpreter in Docker
    There are bugs though
    Running app inside PyCharm was tricky

    View Slide

  80. PyCharm
    Supports Python interpreter in Docker
    There are bugs though
    Running app inside PyCharm was tricky
    node_modules isn't visible to PyCharm

    View Slide

  81. More Tooling
    Plus 666 Other Issues...

    View Slide

  82. Should I Use Docker?

    View Slide

  83. Problems
    Learning curve – you need to learn Docker

    View Slide

  84. Problems
    Learning curve – you need to learn Docker
    You'll also spend time on interesting new issues

    View Slide

  85. Problems
    Learning curve – you need to learn Docker
    You'll also spend time on interesting new issues
    Ongoing problems:
    Tooling

    View Slide

  86. Problems
    Learning curve – you need to learn Docker
    You'll also spend time on interesting new issues
    Ongoing problems:
    Tooling
    Performance outside Linux

    View Slide

  87. Advantages
    Isolation & reproducability

    View Slide

  88. Advantages
    Isolation & reproducability
    Easier to add new services

    View Slide

  89. Advantages
    Isolation & reproducability
    Easier to add new services
    Bundling everything into a single entity

    View Slide

  90. Advantages
    Isolation & reproducability
    Easier to add new services
    Bundling everything into a single entity
    Limiting/monitoring resource use is easier

    View Slide

  91. Advantages
    Isolation & reproducability
    Easier to add new services
    Bundling everything into a single entity
    Limiting/monitoring resource use is easier
    Single-command project setup

    View Slide

  92. It Depends...

    View Slide

  93. It Depends...
    For legacy projects, maybe not

    View Slide

  94. It Depends...
    For legacy projects, maybe not
    But makes more sense for new projects

    View Slide

  95. It Depends...
    For legacy projects, maybe not
    But makes more sense for new projects
    Thorgate open-sourcing our project template

    View Slide

  96. Can You Spare a
    New Tech Point?

    View Slide

  97. Thanks!
    Rivo Laks ⋅ @rivolaks ⋅ rivolaks.com
    CTO at Thorgate ⋅ thorgate.eu
    https://github.com/thorgate/django-project-template

    View Slide