Best practices for building Docker images for Python apps

Docker is very popular among developers and many teams use it to deliver their software. I want to use it too, but my images are large and take forever to build. Does it have to be like that? Is there a better way to do it?

In this presentation, we will go through the whole process of building a docker image for a Python web app. How to select a base image, how to optimize the size, and how to increase the speed of build. We will think about security, logging, and caching and see how to do it all in CI. You will end up with the set of applicable best practices in creating Docker images that you can use in your projects.

Transcript

1. × PyCon SK 2022 Docker + Python Best practices
2. $ whoami Sufio | Senior Python Developer @jozo
3. Do you use Docker?
4. Example app Django + Postgres + React .> Docker github.com/jozo/pycon-sk-2022
5. Dockerfile 1 FROM debian COPY [".", "/app"] RUN apt-get update .& apt-get install -y python3 python3-pip python3-dev libpq-dev RUN cd /app .& pip install -r requirements.txt RUN apt-get install -y nodejs npm RUN cd /app/react-app .& npm install .& npm run build CMD ["gunicorn", ".-workers=2", ".-bind=0.0.0.0", "django_app.wsgi"]
6. Dockerfile 1 FROM debian COPY [".", "/app"] RUN apt-get update .& apt-get install -y python3 python3-pip python3-dev libpq-dev RUN cd /app .& pip install -r requirements.txt RUN apt-get install -y nodejs npm RUN cd /app/react-app .& npm install .& npm run build CMD ["gunicorn", ".-workers=2", ".-bind=0.0.0.0", "django_app.wsgi"] 1.46 GB 2m 10s
7. Size optimization
8. Minimize the number of layers FROM debian COPY [".", "/app"] RUN apt-get update -& apt-get install -y python3 python3-pip python3-dev libpq-dev RUN cd /app -& pip install -r requirements.txt RUN apt-get install -y nodejs npm RUN cd /app/react-app -& npm install -& npm run build CMD ["gunicorn", ".-workers=2", ".-bind=0.0.0.0", "django_app.wsgi"]
9. Minimize the number of layers FROM debian COPY [".", "/app"] RUN apt-get update -& apt-get install -y \ python3 python3-pip python3-dev libpq-dev nodejs npm \ -& cd /app -& pip install -r requirements.txt \ -& cd /app/react-app -& npm install -& npm run build CMD ["gunicorn", ".-workers=2", ".-bind=0.0.0.0", "django_app.wsgi"]
10. Minimize the number of layers FROM debian COPY [".", "/app"] RUN apt-get update -& apt-get install -y \ python3 python3-pip python3-dev libpq-dev nodejs npm \ -& cd /app -& pip install -r requirements.txt \ -& cd /app/react-app -& npm install -& npm run build CMD ["gunicorn", ".-workers=2", ".-bind=0.0.0.0", "django_app.wsgi"] 1.31 GB ~10% less
11. Use small base image FROM debian COPY [".", "/app"] RUN apt-get update .& apt-get install -y \ python3 python3-pip python3-dev libpq-dev nodejs npm \ .& cd /app .& pip install -r requirements.txt \ .& cd /app/react-app .& npm install .& npm run build CMD ["gunicorn", ".-workers=2", ".-bind=0.0.0.0", "django_app.wsgi"]
12. Use small base image FROM debian debian:stable-slim COPY [".", "/app"] RUN apt-get update .& apt-get install -y \ python3 python3-pip python3-dev libpq-dev nodejs npm \ .& cd /app .& pip install -r requirements.txt \ .& cd /app/react-app .& npm install .& npm run build CMD ["gunicorn", ".-workers=2", ".-bind=0.0.0.0", "django_app.wsgi"]
13. Use small base image FROM debian debian:stable-slim python:3-slim COPY [".", "/app"] RUN apt-get update .& apt-get install -y \ python3 python3-pip python3-dev libpq-dev nodejs npm \ .& cd /app .& pip install -r requirements.txt \ .& cd /app/react-app .& npm install .& npm run build CMD ["gunicorn", ".-workers=2", ".-bind=0.0.0.0", "django_app.wsgi"]
14. Use small base image FROM debian debian:stable-slim python:3-slim COPY [".", "/app"] RUN apt-get update .& apt-get install -y \ python3 python3-pip python3-dev libpq-dev nodejs npm \ .& cd /app .& pip install -r requirements.txt \ .& cd /app/react-app .& npm install .& npm run build CMD ["gunicorn", ".-workers=2", ".-bind=0.0.0.0", "django_app.wsgi"] 1.16 GB ~21% less
15. Remove cache and install only what you need FROM python:3-slim COPY [".", "/app"] RUN apt-get update .& apt-get install -y --no-install-recommends \ build-essential libpq-dev nodejs npm \ .& cd /app .& pip install -r requirements.txt \ .& cd /app/react-app .& npm install .& npm run build \ -& rm -rf /var/lib/apt/lists-*
16. Remove cache and install only what you need FROM python:3-slim COPY [".", "/app"] RUN apt-get update .& apt-get install -y --no-install-recommends \ build-essential libpq-dev nodejs npm \ .& cd /app .& pip install -r requirements.txt \ .& cd /app/react-app .& npm install .& npm run build \ -& rm -rf /var/lib/apt/lists-* 831 MB ~43% less
17. Remove cache and install only what you need FROM python:3-slim COPY [".", "/app"] RUN apt-get update .& apt-get install -y .-no-install-recommends \ build-essential libpq-dev nodejs npm \ .& cd /app .& pip install -r requirements.txt \ .& cd /app/react-app .& npm install .& npm run build \ -& apt-get remove -y build-essential \ -& apt-get autoremove -y -& apt-get clean \ .& rm -rf /var/lib/apt/lists.*
18. Remove cache and install only what you need FROM python:3-slim COPY [".", "/app"] RUN apt-get update .& apt-get install -y .-no-install-recommends \ build-essential libpq-dev nodejs npm \ .& cd /app .& pip install -r requirements.txt \ .& cd /app/react-app .& npm install .& npm run build \ -& apt-get remove -y build-essential \ -& apt-get autoremove -y -& apt-get clean \ .& rm -rf /var/lib/apt/lists.* 582 MB ~60% less
19. Use .dockerignore react-app/node_modules/ react-app/build/
20. Use .dockerignore react-app/node_modules/ react-app/build/ 582 MB ~60% less
21. Use multistage build FROM node:18-bullseye-slim AS js-builder COPY ["react-app/", "/app"] RUN cd /app -& npm install -& npm run build FROM python:3-slim COPY --from=js-builder ["/app/build/", "/app/"] COPY [".", "/app"] RUN apt-get update .& . . .
22. Use multistage build FROM node:18-bullseye-slim AS js-builder COPY ["react-app/", "/app"] RUN cd /app -& npm install -& npm run build FROM python:3-slim COPY --from=js-builder ["/app/build/", "/app/"] COPY [".", "/app"] RUN apt-get update .& ... 177 MB ~88% less
23. Use multistage build FROM node:18-bullseye-slim AS js-builder . . . FROM python:3-slim AS py-builder COPY ["requirements.txt", "."] RUN apt-get update .& apt-get install -y .-no-install-recommends \ build-essential libpq-dev \ -& pip wheel --no-cache-dir --no-deps --wheel-dir /opt/wheels -r requirements.txt
24. Use multistage build FROM node:18-bullseye-slim AS js-builder . . . FROM python:3-slim AS py-builder . . . FROM python:3-slim COPY --from=js-builder ["/app/build/", "/app/"] COPY --from=py-builder ["/opt/wheels", "/opt/wheels"] COPY [".", "/app"] RUN pip install --no-cache /opt/wheels-*
25. Use multistage build FROM node:18-bullseye-slim AS js-builder . . . FROM python:3-slim AS py-builder . . . FROM python:3-slim COPY --from=js-builder ["/app/build/", "/app/"] COPY --from=py-builder ["/opt/wheels", "/opt/wheels"] COPY [".", "/app"] RUN pip install --no-cache /opt/wheels-* 172 MB ~88% less
26. 1.46 GB 172 MB -->
27. Dive tool
28. Build time optimization
30. Order layers based on likelihood of change FROM python:3-slim COPY .-from=js-builder ["/app/build/", "/app/"] COPY .-from=py-builder ["/opt/wheels", "/opt/wheels"] COPY [".", "/app"] RUN pip install .-no-cache /opt/wheels.* CMD ["gunicorn", ".-workers=2", ".-bind=0.0.0.0", "django_app.wsgi"]
31. Order layers based on likelihood of change FROM python:3-slim COPY .-from=js-builder ["/app/build/", "/app/"] COPY .-from=py-builder ["/opt/wheels", "/opt/wheels"] RUN pip install .-no-cache /opt/wheels.* COPY [".", "/app"] CMD ["gunicorn", ".-workers=2", ".-bind=0.0.0.0", "django_app.wsgi"]
32. Use BuildKit DOCKER_BUILDKIT=1 docker build .-no-cache -t pycon -f ./Dockerfile-8 .
33. Use cache on CI docker build \ --cache-from image:latest \ --cache-from image:branch-latest
34. Use cache on CI docker build --build-arg BUILDKIT_INLINE_CACHE=1 \ .-cache-from image:latest \ .-cache-from image:branch-latest
35. Upgrade pip RUN pip install --upgrade pip \ .& pip install .-no-cache /opt/wheels.* - Old pip doesn’t support some versions of wheel packages - You get an old version or you have to compile
36. Security & others
37. Base image - Use official images - Pin specific version FROM python:latest ..> FROM python:3.10.7-slim-bullseye
38. Don't use root FROM python:3.10.6-slim-bullseye AS final WORKDIR /app USER python # Copy prince config files COPY ./prince_css/fonts.css /usr/lib/prince/style/fonts.css
39. Upgrade packages during build! Outdated: Upgrading existing packages in a container is an anti pattern RUN apt-get upgrade or RUN apt-get install -y debsecan -& apt-get install \ $(debsecan --suite bullseye --format packages --only-fixed)
40. Security checks in CI docker scan - uses Snyk grype: - local image and db - free - Don’t forget to cache grype’s DB grype $PROD_IMAGE -vv --fail-on medium --only-fixed
41. Other Environment variables: ENV PYTHONUNBUFFERED 1 ENV PYTHONDONTWRITEBYTECODE 1 or RUN python -m compileall -q /app/
42. Other - Use lint .- hadolint - Don't store secrets in images - You can sign and verify images - Add debugging tools: curl, less, top...
44. join.sufio.com github.com/sufio