Better DevOps with MySQL and Docker

Docker is a linux container technology that is taking the world by storm. It has enabled a vibrant ecosystem of open-source and private container images, including dozens of data storage solutions. However, there are still many tricky cases to manage with Docker, such as upgrading containers smoothly, accessing volumes inside and outside the container and bridging network ports from container to host. In this presentation, we survey techniques for managing Docker containers across many data stores and apply the best ones within our own Docker buildfile with the most current practices for database management. We look at MySQL & Docker in the big picture, from development to deployment, operations and maintenance.

Transcript

1. Better DevOps With Docker & MySQL Sunny Gleason Distributed Systems Engineer, SunnyCloud April 15, 2015
2. Who am I? 2 • Sunny Gleason – Distributed Systems Engineer – SunnyCloud, Boston MA • Prior Web Services Work – Amazon – Ning • Focus: Scalable, Reliable Storage Systems for Structured & Unstructured Data
3. What’s this all about? • Why care about Databases & DevOps? • What benefits does Docker provide? • How can we use Docker with MySQL? 3
4. How does DevOps relate to Databases? • Development-Oriented: DB is an empty shell / skeleton schema that enables features to provide new business value • Operations-Oriented: DB is a living organism that provides existing business value 4
5. Development-Focused Perspective 5 Sources: http://support.smartbear.com/ http://www.docbyte.com/en/blog/integrating-gwt-with-spring-and-hibernate • Business/Product defines data needs • Developer maps it into a schema • Developer implements features using whatever app implementation technologies • “Somebody else” takes care of releases, monitoring, escalation
6. Development-Focused Perspective 6 Sources: http://support.smartbear.com/ • Model a schema • Define tables & relations • Make queries easy
7. Development-Focused Perspective 7 Sources: http://www.docbyte.com/en/blog/integrating-gwt-with-spring-and-hibernate • Write the feature • Create Value Objects and
 Data Access Objects • Minimize lines of code
8. Development-Focused Perspective 8 Source: http://www.cumulogic.com/why-a-microservice-architecture-needs-dbaas/ • Create logical/physical
 services • Encapsulate based on
 domain area / vertical • Minimize number of
 systems
9. What’s missing? 9 Source: http://stayhard.typepad.com/development/2013/08/octopus-deployments-in-new-relic.html • Deployment • Release Management
 & Change Management • Monitoring / Alerting • Scaling & Performance
10. Operations-Focused Perspective 10 Source: https://www.mysql.com/products/workbench/ • Instantiate & Deploy
 systems • Manage environments • Hosts & Databases • Storage management • Replication & Backups
11. Operations-Focused Perspective 11 Source: http://www.slideshare.net/aca_it/modularity-ddd • Coordinate schema
 migrations • Manage replication • Minimize downtime • Maximize performance
12. Operations-Focused Perspective 12 Source: http://ronaldbradford.com/images/blog/newrelic-summary1.png • Create views • Monitor & Alert on
 key metrics • Inform scaling &
 perf optimization
13. Operations-Focused Perspective 13 Source: https://databasesincloud.wordpress.com/2011/11/12/scaling-oracle-databases-on-amazon-rds/ • Do initial scale plan • Monitor capacity
 constraints • Perform capacity
 updates • Continuous feedback
 loop
14. Where does DevOps fit in? 14 Source: http://blog.appdynamics.com/devops/devops-scares-me-part-2/ • “Developer folks” should be
 more operations-aware • “Systems folks” should be
 more development-aware • Goal: Everyone should be more
 responsive to the business & market
15. How can Docker help? 15 • Docker is not DevOps • Docker will not do your laundry • Docker will let you create standard images • Docker will let you deploy & run versioned images • Docker is lightweight enough for developers to use • Docker is powerful enough to solve many needs • Docker will not coordinate your systems
16. What is the core idea of Docker? 16 Traditional Virtualization
 (Xen, VMWare, etc.) Source: https://www.docker.com/
17. What does Docker provide? 17 • Container System • Container Format • Container Buildfiles • Container Versioning • Repository API • Deployment Mechanism • Container Runtime • Virtualization via libvirt • Container Library • Cross-platform support
18. How do we work with Docker? 18 Source: http://blog.octo.com/en/docker-registry-first-steps/
19. Docker Gaps / Gotchas 19 • Still maturing • Rapid release cycle • Network configuration • No Easy Introspection • No “dom0” standard • Cross-platform is new • Security • Troubleshooting • I/O-Intensive Deploy • No Service Discovery • Differing Integrations • Orchestration
20. What’s Docker’s value proposition? 20 • Ability to collaborate & version standard images • Easier deployment across environments / stages • Developer-friendly installation & usage • Operations-friendly controls & management
21. Getting Started with Docker 21 • Check out: http://docs.docker.com/installation/ • Install Docker on Linux using something like:
 $ wget -qO- https://get.docker.com/ | sh • (Install Boot2docker or Kitematic on OS X) • Deploy Containers using “docker run” • A somewhat-authoritative MySQL image is at:
 https://registry.hub.docker.com/_/mysql/
22. Docker Summed Up in 1 Command 22 
 docker run --name some-mysql 
 -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:tag • Downloads image with version “mysql:tag” • Creates a new container called “some-mysql” • Starts the container process using the default endpoint
23. First Commands With Docker 23 • docker pull : downloads an image (makes available for use) • docker run : creates a container from an image • docker start / stop / restart : container process control • docker save : saves container state as a tar file • docker load : loads container state from a tar file • docker build : builds a new image from a Dockerfile • docker commit : creates a new revision of an image from a container • docker tag : associates a tag with a commit (think: git) • docker attach : attach console to running container • docker logs : fetch the logs of a container
24. FROM debian:wheezy 
 RUN groupadd -r mysql && useradd -r -g mysql mysql RUN apt-get update && apt-get install -y perl --no-install-recommends 
 && rm -rf /var/lib/apt/lists/* RUN apt-key adv --keyserver pool.sks-keyservers.net \
 --recv-keys A4A9406876FCBD3C456770C88C718D3B5072E1F5
 ENV MYSQL_MAJOR 5.7 ENV MYSQL_VERSION 5.7.7-rc 
 RUN echo "deb http://repo.mysql.com/apt/debian/ wheezy mysql-$ {MYSQL_MAJOR}-dmr"
 > /etc/apt/sources.list.d/mysql.list 
 ...
 MySQL 5.7 Dockerfile (part 1) 24 Source: https://raw.githubusercontent.com/docker-library/mysql/master/5.7/Dockerfile
25. MySQL 5.7 Dockerfile (part 2) 25 Source: https://raw.githubusercontent.com/docker-library/mysql/master/5.7/Dockerfile ... 
 # comment out a few problematic configuration values RUN sed -Ei 's/^(bind-address|log)/#&/' /etc/mysql/my.cnf VOLUME /var/lib/mysql COPY docker-entrypoint.sh /entrypoint.sh ENTRYPOINT ["/entrypoint.sh"] EXPOSE 3306 CMD ["mysqld"]
26. Keep in mind: file locations • Docker stores files in /var/lib/docker • Containers live in “containers” dir • Images live in “graph” • You can inspect/modify file systems
 (at own peril) 26
27. Keep in mind: users & permissions • Docker containers tend to run entry points as root • In mysql case, uses mysql user • User-level security is not 100% 27
28. Keep in mind: externalize volumes • Docker containers are self-contained by default • External volumes can be mounted using the
 -v /source/external:/source/internal
 argument to docker run • This is important for: data dir, log dir, possibly config dir 28
29. Keep in mind: network ports • Each docker container has its own virtual ip • Ports are mapped from internal to external using 
 -P 3306:3333
 argument to docker run • There will be port conflicts to manage 29
30. Keep in mind: entry points • Each docker container starts with a default entry point • This is usually the only process running in the container • To do more, need to use a supervisor process or / bin/bash as entry point • This affects your ability to inspect inside the container using the default attach command 30
31. Keep in mind: storage management • The default docker directory is
 /var/lib/docker • Consider mounting /var/lib/docker as an independent device using /etc/fstab or configure a docker directory for the docker daemon • It’s tricky to analyze the storage requirements of images since revisions are shared 31
32. What does this mean for development? • Ability to use Docker on Linux, OS X • Provides mechanism for image versioning and easier provisioning / running of containers • Development environments are less unique, more throwaway • Ability to run multiple container instances 32
33. What does this mean for staging/test? • Staging and test can be closer to development • Staging and test environments are less unique, more throwaway • Ability to run multiple container instances • Point in time activation / deactivation of containers 33
34. What does this mean for production? • Production can be closer to development • Production environments are more throwaway • Ability to run multiple container instances • Ability to deploy software updates more easily on same hardware (different containers using same data volume) 34
35. What’s next? • Better cross-platform support • Better service discovery & network config • Better orchestration • Wider range of integrations (AWS, Google, …) 35
36. What’s going to be tricky for a while? • Security • Performance & process isolation • Resource management • “dom0” Provisioning • Network configuration 36
37. Thank You! 37