Docker Lessons from Real-World Projects

Presented at DjangoCon Europe 2017

Explores how to use Docker for complex, real-world Django applications.

Transcript

1. Docker Lessons from Real-World Projects Rivo Laks ⋅ Thorgate 2017-04-05
2. The Problem
3. The Problem At Thorgate, we host dozens of our projects
4. The Problem At Thorgate, we host dozens of our projects A single server runs multiple projects
5. The Problem At Thorgate, we host dozens of our projects A single server runs multiple projects Dependency isolation became an issue
6. So we thought – Docker!?
7. Why Docker? provides isolation
8. Why Docker? provides isolation exotic services easier to use
9. Why Docker? provides isolation exotic services easier to use project bundled into a single image
10. How to Get There? Lots of online tutorials for making rst steps Practical experience with bigger projects is much more scarce
11. Wait, What's Docker?
15. Your App on Docker
16. Application Layout Few assumptions:
17. Application Layout Few assumptions: Django, Node.js, database, other services
18. Application Layout Few assumptions: Django, Node.js, database, other services Single-server apps only
19. Application Layout Few assumptions: Django, Node.js, database, other services Single-server apps only Looking at production rst
21. The Django Service
22. The Django Service Start with o cial python image
23. The Django Service Start with o cial python image Install Python requirements
24. The Django Service Start with o cial python image Install Python requirements Add application source code
25. The Django Service Start with o cial python image Install Python requirements Add application source code Use Gunicorn to serve HTTP
26. The Django Service Start with o cial python image Install Python requirements Add application source code Use Gunicorn to serve HTTP Volumes are used for data dirs This includes static assets, media and logs
27. The Node Service Bundles JS + CSS using Webpack
28. The Node Service Bundles JS + CSS using Webpack Isn't run continuously, just once during deploy
29. The Node Service Bundles JS + CSS using Webpack Isn't run continuously, just once during deploy Basically o cial Node image + our requirements
30. The Node Service Bundles JS + CSS using Webpack Isn't run continuously, just once during deploy Basically o cial Node image + our requirements Volumes for JS/CSS source dirs
31. Other Services Redis, Celery, etc
32. Other Services Redis, Celery, etc Can often use o cial image out-of-the-box
33. Other Services Redis, Celery, etc Can often use o cial image out-of-the-box Volumes for persistent data
34. Putting it All Together Our application has multiple services working in unison, communicating via network
35. Putting it All Together Our application has multiple services working in unison, communicating via network Docker Compose is “a tool for de ning and running multi-container Docker applications”
36. Putting it All Together Our application has multiple services working in unison, communicating via network Docker Compose is “a tool for de ning and running multi-container Docker applications” de ne & control multiple containers at once
37. Putting it All Together Our application has multiple services working in unison, communicating via network Docker Compose is “a tool for de ning and running multi-container Docker applications” de ne & control multiple containers at once easier networking among single app's services
39. Docker Compose – Examples docker-compose build
40. Docker Compose – Examples docker-compose build docker-compose up
41. Docker Compose – Examples docker-compose build docker-compose up docker-compose logs
42. Docker Compose – Examples docker-compose build docker-compose up docker-compose logs docker-compose run django ./manage.py migrate
43. The Servers Almost nothing but Docker
44. The Servers Almost nothing but Docker Postgres and Nginx as well, since we chose to leave those global
45. Deploys
46. Deploys docker-compose build
47. Deploys docker-compose build docker-compose run --rm node npm run build
48. Deploys docker-compose build docker-compose run --rm node npm run build docker-compose run --rm django ./manage.py collectstatic docker-compose run --rm django ./manage.py migrate
49. Deploys docker-compose build docker-compose run --rm node npm run build docker-compose run --rm django ./manage.py collectstatic docker-compose run --rm django ./manage.py migrate docker-compose up -d
50. Development Environment
51. Dev vs Production Should ideally be very similar, but some di erences are useful
52. Dev vs Production Should ideally be very similar, but some di erences are useful Commands are slightly di erent
53. Dev vs Production Should ideally be very similar, but some di erences are useful Commands are slightly di erent Sources not baked into images
54. Dev vs Production Should ideally be very similar, but some di erences are useful Commands are slightly di erent Sources not baked into images Postgres is run locally, along other services
55. Dev vs Production Should ideally be very similar, but some di erences are useful Commands are slightly di erent Sources not baked into images Postgres is run locally, along other services Separate Docker Compose con g le
56. Django App dir is mounted as volume, not baked into image
57. Django App dir is mounted as volume, not baked into image Python dependencies still part of the image
58. Django App dir is mounted as volume, not baked into image Python dependencies still part of the image Django's runserver instead of Gunicorn
59. Django App dir is mounted as volume, not baked into image Python dependencies still part of the image Django's runserver instead of Gunicorn Separate Dockerfile
60. Node Image contains libraries but not our sources (same as production)
61. Node Image contains libraries but not our sources (same as production) Webpack watch mode, similar to Django's runserver
62. Other Services Similar to production Postgres is run locally
63. Other Services Similar to production Postgres is run locally Project-speci c volumes for data
64. Quirks aka The Death by a Thousand Papercuts
65. Commands docker-compose run --rm django ./manage.py runserver docker-compose run --rm django ./manage.py migrate
66. Commands docker-compose run --rm django ./manage.py runserver docker-compose run --rm django ./manage.py migrate Make le FTW! make docker make migrate
67. Commands docker-compose run --rm django ./manage.py runserver docker-compose run --rm django ./manage.py migrate Make le FTW! make docker make migrate Bonus: make setup
68. Service Discovery How to route HTTP requests to correct Django container
69. Service Discovery How to route HTTP requests to correct Django container Ran Nginx in container, used Docker's built-in routing
70. Service Discovery How to route HTTP requests to correct Django container Ran Nginx in container, used Docker's built-in routing But container IP can change, Nginx does DNS lookup only once
71. Service Discovery How to route HTTP requests to correct Django container Ran Nginx in container, used Docker's built-in routing But container IP can change, Nginx does DNS lookup only once Created a service monitoring Docker events and reloading Nginx
72. Waiting for Related Services Need to wait for Postgres to be available for Django
73. Waiting for Related Services Need to wait for Postgres to be available for Django wait-for-it.sh scripts
74. Performance on non-Linux OSs On macOS / Windows, Docker containers run in VM
75. Performance on non-Linux OSs On macOS / Windows, Docker containers run in VM Filesystem performance su ers
76. Performance on non-Linux OSs On macOS / Windows, Docker containers run in VM Filesystem performance su ers localhost isn't localhost anymore
77. PyCharm Supports Python interpreter in Docker
78. PyCharm Supports Python interpreter in Docker There are bugs though
79. PyCharm Supports Python interpreter in Docker There are bugs though Running app inside PyCharm was tricky
80. PyCharm Supports Python interpreter in Docker There are bugs though Running app inside PyCharm was tricky node_modules isn't visible to PyCharm
81. More Tooling Plus 666 Other Issues...
82. Should I Use Docker?
83. Problems Learning curve – you need to learn Docker
84. Problems Learning curve – you need to learn Docker You'll also spend time on interesting new issues
85. Problems Learning curve – you need to learn Docker You'll also spend time on interesting new issues Ongoing problems: Tooling
86. Problems Learning curve – you need to learn Docker You'll also spend time on interesting new issues Ongoing problems: Tooling Performance outside Linux
87. Advantages Isolation & reproducability
88. Advantages Isolation & reproducability Easier to add new services
89. Advantages Isolation & reproducability Easier to add new services Bundling everything into a single entity
90. Advantages Isolation & reproducability Easier to add new services Bundling everything into a single entity Limiting/monitoring resource use is easier
91. Advantages Isolation & reproducability Easier to add new services Bundling everything into a single entity Limiting/monitoring resource use is easier Single-command project setup
92. It Depends...
93. It Depends... For legacy projects, maybe not
94. It Depends... For legacy projects, maybe not But makes more sense for new projects
95. It Depends... For legacy projects, maybe not But makes more sense for new projects Thorgate open-sourcing our project template
96. Can You Spare a New Tech Point?
97. Thanks! Rivo Laks ⋅ @rivolaks ⋅ rivolaks.com CTO at Thorgate ⋅ thorgate.eu https://github.com/thorgate/django-project-template