m-cuzzoli-g-fanelli-managing-docker-with-python-number-domotz-at-pycon6

Looking at the most successful open source projects we have been seriously impressed by the importance of Docker: it is positively revolutionizing the DevOps industry. Docker is a valid alternative to the traditional virtual machines and it is redefining the IT service deployment.

Thanks to the available Docker APIs, we have witnessed a remarkable proliferation of tools based on the same APIs to efficiently manage virtual infrastructures: a place of honor, among of all of these, is reserved to the libraries and tools written in Python.

The talk gives an overview about the Python modules that allow you to interact with Docker and it is divided into two parts:

· The first part provides an introduction to Docker by illustrating a typical use case: a network of micro-services aimed to manage a web application.

· The second part goes into detail on how to write Python code which will be able to orchestrate Docker images and containers.

Transcript

1. Managing Docker with Python Massimiliano Cuzzoli & Giancarlo Fanelli
2. Who We Are • Research & Development at Domotz • Internet of Things startup founded in 2014 • Offices in London (UK), Pisa (Italy) and Norwalk (USA) We Are Hiring!!!
3. What We Will Cover Today • Introduction to Docker • A sample of Dockerized application • Python Tools for Docker • An Interactive Demo • Q&A Section
4. Virtualization Matters • Efficient HW Utilization • Improved Disaster Recovery • Faster Provisioning • Application Isolation • Cloud-Ready Appliances
5. Traditional VM vs Docker Hardware Host OS VM Monitor Guest OS Bin Deps Your App 1 Guest OS Bin Deps Your App 2 Hardware Host OS (Linux) cgroups, namespaces Your App 1 Bin Deps D o c k e r Your App 2 Bin Deps
6. What Docker Is • Linux Containers for Humans: ◦ Easy Packaging ◦ Easy Deployment ◦ Easy Versioning Hardware Host OS (Linux) cgroups, namespaces Your App 1 Bin Deps D o c k e r Your App 2 Bin Deps
7. Basic Concepts: Images & Containers • Docker Image ◦ Folders / Files of your appliance ◦ The configuration to launch your appliance ◦ Read Only Template • Docker Container ◦ Read/Write instance of the Image where your appliance runs Note Docker Images can be built extending an existing one
8. Docker Core Components • Docker Daemon ◦ Daemon that manages your images/containers • Docker CLI ◦ Command line for interacting with the daemon • Docker HUB ◦ Repository of docker images (image versioning)
9. ‘Dockerize’ Your Application
10. Music Awards: The Project • Sample App: Voting System for Music Awards • A Python Web App based on Bottle PUT http://<webserver>/music_award/<artist> GET http://<webserver>/music_award/<artist>
11. Music Awards: The Code redis_conn = StrictRedis('redis.local') app = application = Bottle() @app.put('/music_award/<artist>') def artist_voting(artist): response.content_type = 'application/json' redis_conn.incr('music_award/{}'.format(artist)) return json.dumps({'success': True}) @app.get('/music_award/<artist>') def artist_page(artist): response.content_type = 'application/json' counter = redis_conn.get('music_award/{}'.format(artist)) return json.dumps({'artist': artist, 'votes': counter or 0})
12. Music Awards: Main Components • Service Components Python WebApp (Bottle) Web Server (NGINX) Data Persistency (Redis)
13. Music Awards: Main Components • Micro Services Approach Python WebApp (Bottle) Docker Container 1 Web Server (NGINX) Data Persistency (Redis) Docker Container 2 Docker Container 3 1. Build your Docker Image (write your Dockerfile) 2. Start your Container Dockerizing Your Web App
14. Docker Container 2 Python WebApp (Bottle) Docker Container 1 Web Server (NGINX) Docker Container 3 Data Persistency (Redis) Docker Container 1 Web Server (NGINX) Data Persistency (Redis) Python WebApp (Bottle) Docker Container 2 Docker Container 3 FROM ubuntu:trusty RUN apt-get update RUN apt-get install -y python python-dev python-setuptools RUN easy_install pip RUN mkdir /opt/my_webapp COPY app/* /opt/my_webapp/ # install dependencies RUN pip install -r /opt/my_webapp/requirements.txt EXPOSE 8010 CMD ["/usr/local/bin/uwsgi", "--ini", /opt/my_webapp/simple_webapp.ini"] Write Your Dockerfile Python WebApp
15. # No Dockerfile required # We use the standard Docker Image for Redis FROM dockerfile/nginx COPY music_award.conf /etc/nginx/sites-enabled/ EXPOSE 81 Docker Container 2 Python WebApp (Bottle) Docker Container 1 Web Server (NGINX) Docker Container 3 Data Persistency (Redis) Write Your Dockerfile
16. Run Your Containers $> docker build -t pycon/sample_webapp ./webapp $> docker build -t pycon/nginx ./nginx $> docker run --name redis_1 redis:latest $> docker run --name webapp_1 --link redis_1 pycon/sample_webapp $> docker run --name nginx_1 --link webapp_1 -p 6081:81 pycon/nginx
17. Python Tools For Docker
18. Python Tools for Docker: Fig • Define your service set within a single configuration file • Start & Stop your services with a single command Docker Compose
19. Docker Compose nginx: build: ./nginx links: - webapp:webapp.local ports: - "6081:81" redis: image: redis expose: - "6379" webapp: build: ./webapp links: - redis:redis.local expose: - "8010" • Single file with YAML syntax • Single command to start all of your components $ > docker-compose up
20. Python Tools for Docker Docker Compose Docker Py
21. Docker Py Docker Daemon Remote API unix://<path> tcp://<host:port> Container 1 Container k Python Application Docker Py
22. Docker Py: Notable Features • Information Gathering • Image Building • Container Orchestration • Docker HUB interaction
23. Docker Py: Information Gathering >>> import docker >>> d_client = docker.Client(base_url='unix://var/run/docker.sock') >>> d_client.containers() [{u'Command': u'nginx', u'Created': 1427641784, u'Id': u'eec1e895c09e0cfe15bf1a1a1fab69d25d20afec90c78d8e8dbea29b088417ed', u'Image': u'dockerimages_nginx:latest', u'Names': [u'/dockerimages_nginx_1'], u'Ports': [{u'IP': u'0.0.0.0', u'PrivatePort': 81, u'PublicPort': 6081, u'Type': u'tcp'}, {u'PrivatePort': 443, u'Type': u'tcp'}, {u'PrivatePort': 80, u'Type': u'tcp'}], u'Status': u'Up 14 minutes'}]
24. Docker API is (almost) REST HTTP/1.1 101 UPGRADED Content-Type: application/vnd.docker.raw-stream Connection: Upgrade Upgrade: tcp {{ STREAM }} • Some information are sent in streaming mode • The communication channel is ‘upgraded’ to TCP • Information are retrieved via a raw socket receive
25. Docker Py: Information Gathering >>> logs_gen = docker.logs(container['Id'], stream=True) >>> logs_gen <generator object _multiplexed_response_stream_helper at 0x7feefad98e10> >>> logs_gen.next() '[uWSGI] getting INI configuration from /opt/my_webapp/simple_webapp.ini'
26. Docker Py: Container Orchestration >>> import docker >>> d_client = docker.Client(base_url='unix://var/run/docker.sock') >>> container = d_client.create_container(image=DOCKER_IMAGE_NAME, command='nginx', mem_limit='512m', cpu_shares=256, detach=True) >>> d_client.start(container=container['Id']) >>> d_client.stop(container=container.['Id'], timeout=30)
27. Docker Py An Interactive Sample
28. An Interactive Sample Web Server HTTP Requests access.log Container 1 Container N
29. Thank You • Contacts massimiliano@domotz.com giancarlo@domotz.com • About Domotz @domotz http://www.domotz.com