Dorian Pula - Pythons in A Container - Lessons Learned Dockerizing Python Micro-Services

Micro-services and Docker are all the rage for developing scalable systems. But what challenges will you face when developing and deploying Python apps using Docker to production? This talk goes into the real-life lessons learned from creating, deploying and scaling Dockerized Python applications.

https://us.pycon.org/2016/schedule/presentation/2096/

Transcript

1. PYTHONS IN A CONTAINER LESSONS LEARNED DOCKERIZING PYTHON MICROSERVICES... ...THE HARD WAY    Presented by / Dorian Puła @dorianpula
2. INTRODUCTION 
3. WHO AM I? So ware Development Engineer @  Develop eCommerce platform for Loyalty Programs (Buy, Gi + Transfer points)  Flask REST APIs + Apps  Dockerized microservices Open Source  - Yet Another CMS  Contributed to Fabric, Ansible & core Python  Ansible roles for NGINX, UWSGI, NodeJS and Supervisor Points Rookeries
4. WHAT IS THIS TALK ABOUT? Lessons learned using Docker for Flask REST API and apps. Incorporating various tools that Docker and docker-compose provide for better DevOps workflow. The usefulness of unlearning some accepted patterns in Python development, when working with Docker.
5. WHAT IS THIS TALK NOT ABOUT? An introduction to basic Docker or WSGI apps. Docker Machine (cool as it is). Advanced Docker wizardery. See Dockercon next week for that. An exposé on why you must or must not use Docker.
6. MICROSERVICES + DOCKER 
7. EXAMPLE APP + API - POINTS FOR PYTHONISTAS Imagine having to build an app for a new hypothetical loyalty program for sprint contributers at PyCon. Earn points per commit or issue resolved. Redeem points for essential sprint goods. (e.g. coffee, poptarts or dogecoin.) Has the following components: REST API Frontend App Redemption of Points User + Project Registration/Linking Database
8. WHY A MICROSERVICES ARCHITECTURE? Imagine implementing said example using a microservices architecture, with multiple services built by multiple teams. Benefits:  Smaller less complex codebases.  Enable independence between codebases & teams.  More flexible scaling schemes (tech & organizational). Drawbacks:  Distributed codebases harder to infer, and may contain implicit inter-service dependencies.  More complex orchestration, monitoring & provisioning.
9. EXAMPLE ARCHITECTURE Points App + API Redeem Service User + Project Registry Service Database Database Dogecoin Database Bitbucket
10. WHY USE DOCKER?  Containers vs. Virtual machines Containers lighter in memory and processing than VMs. Isolated user-space instances vs. machine emulation. Docker uses cached/immutable layered file systems.  Tooling for Managing Containers Quick spin up of container/environments. Easily create, share and publish images to registries. Unified workflow that replaces other tools: e.g. chroot jails, LXC, Vagrant, etc.
11. DEVELOPMENT AND TESTING 
12. DOCKER COMPOSE Specify with docker-compose.yaml... p o i n t s _ a p p : b u i l d : . p o r t s : ­ " 5 0 0 0 : 5 0 0 0 " e n v i r o n m e n t : ­ A P I _ K E Y = M Y _ S U P E R _ S E C R E T _ K E Y h o s t n a m e : a p p l i n k s : " c o u c h d b : c o u c h " c o u c h d b : i m a g e : c o u c h d b p o r t s : ­ " 5 9 8 4 : 5 9 8 4 " v o l u m e s : ­ d a t a : / u s r / l o c a l / v a r / l i b / c o u c h d b o t h e r _ s e r v i c e s : . . . ...and start up with: d o c k e r ­ c o m p o s e u p
13. DOCKER WORKFLOW Docker + Compose replaces a Vagrant + VM workflow vagrant up + vagrant ssh + run $app_command  docker run $app_command vagrant halt  docker stop vagrant status  docker ps vagrant provision  docker build vagrant destroy  docker stop + docker rm vagrant box list, remove  docker images, docker rmi
14. BUILDING GOOD DOCKER IMAGES  Sample Dockerfile F R O M u b u n t u : 1 6 . 0 4 R U N a p t ­ g e t u p d a t e & & a p t ­ g e t i n s t a l l ­ y p y t h o n p y t h o n ­ d e v g c c \ p y t h o n ­ p i p p y t h o n ­ s e t u p t o o l s A D D w s g i _ a p p / a p p W O R K D I R / a p p R U N p i p i n s t a l l ­ r r e q u i r e m e n t s . t x t & & p i p i n s t a l l u w s g i C M D u w s g i ­ ­ h t t p : 5 0 0 0 ­ ­ m a s t e r ­ ­ p r o c e s s e s 4 ­ ­ w s g i ­ f i l e a p p _ w s g i . p y # C M D p y t h o n a p p _ w s g i . p y E X P O S E 5 0 0 0 Each step in a Dockerfile can create a new layer in filesystem. Minimize steps number of separate RUN steps. Try to make layers cacheable: Cached layer reused if no checksum change in source. Use base images for heavily repeated steps. See ONBUILD command for making dynamic base images. Expose ports and volumes to document image.
15. PYTHON AND WSGI APPS  Web Servers Don't run a web server on your container. Use an external proxy or container instead. Just run WSGI apps using a WSGI app server: uWSGI Gunicorn  Virtualenvs Don't use virtualenvs inside Docker containers! Install directly into the system Python site packages.
16. DEBUGGING CONTAINERS Want a minimal image, so no SSH daemon... ...so how do we debug a running container?  Run Bash (or other command) on a Running Service  Inspecting a Service's Logs (Standard Out & Error)  Inspecting a Running Container's Setup d o c k e r ­ c o m p o s e e x e c $ S E R V I C E _ N A M E / b i n / b a s h d o c k e r ­ c o m p o s e l o g s $ S E R V I C E _ N A M E d o c k e r i n s p e c t $ C O N T A I N E R _ I D > . . . d o c k e r i n s p e c t ­ ­ f o r m a t ' { { j s o n . C o n f i g . E x p o s e d P o r t s } } ' \ $ C O N T A I N E R _ I D > { " 5 0 0 0 / t c p " : { } }
17. PERSISTANCE, CONFIGS & PROCESSES  Volume Maps Changes to container lost a er container destroyed. Volume maps to external host folder for persistence. Another pattern is using separate Docker data containers.  Configuration Prefer using environment variables for configuration. Volume mapped configs maybe a warning sign of a overly complex setup or a config in need of refactoring.  Managing Processes Use supervisord or runit to control multiple processes. Consider refactoring containers to not need that.
18. TESTING + TOOLING  Testing Docker adds consistency in your CI environments! Simple setup for a Docker host. Control over what is in container = Repeatable workflow and simpler test environment. Cloud-based CI options with Docker support out there.  Tooling Docker tool defaults, options, and internal API can radically from version to version. Don't build your own tooling! If you can avoid it... docker-py: a Python client library for working with Docker*
19. DEPLOYMENT AND SCALING 
20. EXAMPLE ARCHITECTURE Points App + API Redeem Service User + Project Registry Service Database Database Dogecoin Database Bitbucket
21. EXAMPLE PROD ENVIRONMENT Datacenter 2 Datacenter 2 LoadBalancer User + Project Registry Service Redeem Service Points App + API Datacenter 1 Datacenter 1 Points App + API Points App + API Redeem Service User + Project Registry Service Database Database Database LoadBalancer LoadBalancer User + Project Registry Service Redeem Service Points App + API Points App + API Points App + API Redeem Service User + Project Registry Service Database Database Database LoadBalancer LoadBalancer
22. SETTING UP A CLOUD Looks like you're trying to build a cloud of microservices...  Load Balancing + Network Topology: e.g. HAProxy & Nginx, etc.  Provisioning: Automated, repeatable setup for non-Docker systems. e.g. Ansible, Puppet & Salt.  Monitoring: Look at app health, app behaviour & system resources. e.g. Nagios, Pingdom & New Relic.  Logging: Aggregate various logs and correlate events. e.g. Splunk.
23. CLOUD INFRASTRUCTURE  Managing cloud infrastructure is hard!  Need tooling and automation for all that stuff.  Don't build your own tool unless you want to support it to end of time. (Unless you're a cloud tech vendor.)  Consider using one of these instead: Docker Swarm Kubernetes OpenStack Magnum CoreOS Fleet
24. LESSONS LEARNED 
25. LESSONS LEARNED Microservices and Docker can improve building and deploying complex systems. But neither is a cure-all. Good development & deployment processes matter. Docker has a decent workflow to help shape those processes. Expect lots of additional infrastructure around microservices. Avoid building your own tooling. Use Docker containers to do effective isolation. Good app design goes a long way.
26. RESOURCES Jared Kerim's Django Docker template: 12 Factor apps: Rookeries - Dockerized Workflow Example: https://github.com/jaredkerim/django-docker-compose http://12factor.net/ https://bitbucket.org/dorianpula/rookeries/ (docker_compose_workflow branch)
27. THANK YOU!  Twitter - @dorianpula  WWW - http://dorianpula.ca/ ANY QUESTIONS? GO FORTH AND BUILD AWESOME STUFF!!!