Docker @Viadeo

The whole Viadeo's technical stack on one dev station using Docker
by Nicolas Colomer and Quention Suire, Software Engineers at Viadeo

1. Docker @ Viadeo Docker Paris - 25/02/2015
2. Who we are... Nicolas Colomer @n_colomer Software Engineer @Viadeo, DevOps, (Big) Data, Geomatics, Electronics and OSS Quentin Suire @Kuhess Software Engineer @ Viadeo, DevOps, Machine Learning, Robotic & Science
3. - Problematic - Containerization - Industrialization - Feedbacks - So what? Agenda
4. Mention supplémentaire Problematic Houston, we have a problem!
5. Meet Viadeo’s architecture network-graph MySQL platform memcached HBase elasticsearch 0.20.6 RabbitMQ consult-profile legacy middle-end (web) graph-api JS client middle-end (mobile) mobile apps elasticsearch 1.1.2 Spark jobs (over Mesos) Hadoop thumbor BATCH AWS + SAAS CLOUD DBs BACK FRONT :)
6. commits are everywhere in the stack! we develop new product features, not tech stuff share the same environment with everyone need to run integration tests locally Full-stack developments
7. each dev install his env based on his needs 1 backend = n README (install, conf, etc...) ideally, all dependencies are installed locally... ... or defaults to demo/integ environments The stone age
8. concurrency on mutualized environments README are painful to maintain dev station installation and configuration is hell! backing services are stateful test results may be unpredictable Then comes the crap
9. each build triggers its dependent backends cheap (well, not really) POC proved it runs! despite: - slower tests - unstable results (leak) - dirty (java triggers shell + compatibility) - backend specific Sweep the past away
10. Docker to the rescue! 1 backend = 1 container (+ initialization scripts) system & service configuration + initial state
11. Mention supplémentaire Containerization Go! Go! Docker!
12. What is containerization? Back end Service Deps/conf port
13. Example with MySQL Application MySQL port
14. ★ A back end image must work out of the box on localhost. ★ Connect to a back end image must respect the back end conventions. ★ A back end must not contain "homemade" data. Golden Rules
15. Port is 3306 User is root Password is "" Configure with “defaults”
16. EXAMPLE : UTF-8 ENCODING /etc/mysql/my.cnf Add some custom configuration ... [mysqld] character-set-server = utf8 collation-server = utf8_swedish_ci character-set-client = utf8 ...
17. FROM google/debian:wheezy # Install MySQL server 5.5 RUN apt-get update && \ apt-get install -y mysql-server-5.5 && \ apt-get clean # Configure root user ADD ./configure_mysql.sh /tmp/configure_mysql.sh RUN /tmp/configure_mysql.sh # MySQL custom configuration ADD ./my.cnf /etc/mysql/my.cnf EXPOSE 3306 CMD ["/usr/bin/mysqld_safe"] Resulting Dockerfile
18. Let’s use it :) mysql -u root CONNECT TO THE SERVICE docker build -t viadeo/mysql . docker run -p 3306:3306 viadeo/mysql BUILD THE IMAGE AND RUN THE CONTAINER
19. - simplicity - speed (LXC) - maintainability (Dockerfile ≈ sh) - known starting state Why Docker?
20. Mention supplémentaire Industrialization 1 + 1 + … + 1 = N
21. 1 APP with 1 SERVICE But how to scale? 1 APP with N SERVICES ? Scalability of services
22. Scale with command-line? docker run [opt...] viadeo/service1 docker run [opt...] viadeo/service2 … docker run [opt...] viadeo/serviceN Hard to read and maintain! How to deal with dependencies?
23. Use Orchestration! name: local ships: local: ip: 127.0.0.1 services: service1: image: "viadeo/service1" instances: service1: { … } [ … ] serviceN: image: "viadeo/serviceN" requires: [serviceX] instances: serviceN: { … } Maestro
24. Images are shared using a Docker registry. Use a Docker Registry We use Quay.io on SaaS REGISTRY
25. Push Tested Images Git Repository Github Docker Registry Quay.io Continuous Integration CircleCI
26. Basic test with Elasticsearch How to test your images? Elasticsearch 9200 docker build -t viadeo/elasticsearch . docker run -d -p 9200:9200 viadeo/elasticsearch # Test that Elasticsearch is alive curl -s "http://localhost:9200" curl localhost:9200
27. Advanced test with Memcached How to test your images? Memcached 11211 MEMCACHED Client Memcached Tester
28. Advanced test with Memcached How to test your images? docker-memcached ├── circle.yml ├── Dockerfile ├── README.md ├── run_test.sh └── test ├── Dockerfile ├── main.py └── requirements.txt
29. Advanced test with Memcached test/main.py How to test your images? def test_memcached(self): host = os.getenv('MEMCACHED_PORT_11211_TCP_ADDR') port = int(os.getenv('MEMCACHED_PORT_11211_TCP_PORT')) address = "%s:%d" % (host, port) client = memcache.Client([address], debug=0) client.set("some_key", "Some value") value = client.get("some_key") self.assertEqual("Some value", value)
30. Advanced test with Memcached run_test.sh How to test your images? # Build images docker build -t viadeo/memcached . docker build -t memcached_tester test # Run main container docker run -d --name=memcached viadeo/memcached # Wait Memcached to launch sleep 1 # Run tester container docker run --rm --link=memcached:memcached memcached_tester
31. Continuous Integration Cycle App Integration Dockerfile Integration Docker Registry Continuous Integration Git Repository Dev Env
32. Mention supplémentaire Feedbacks You know, man!
33. Network and DNS resolution Zookeeper
34. SaaS downtimes The Murphy’s Law
35. CircleCI VM setup The Evil Twin
36. Multi-OS integration Wrong way!
37. Mention supplémentaire So what? The takeaway
38. Where we are? network-graph MySQL platform memcached HBase elasticsearch 0.20.6 RabbitMQ consult-profile legacy middle-end (web) graph-api JS client middle-end (mobile) mobile apps elasticsearch 1.1.2 Spark jobs (over Mesos) Hadoop thumbor AWS + SAAS Containerized!
39. ISO and multi-OS environments for our devs easy to deploy and auto-configured ╭─me@v581 /tmp ╰─$ python -m maestro start # INSTANCE SERVICE SHIP CONTAINER STATUS 1. elasticsearch elasticsearch local latest:984f444 up for 4d11h20m 2. elasticsearch_0206 elasticsearch_0206 local 0.20.6:27ebcb9 up for 4d11h20m 3. hbase hbase local latest:c9d87d6 up for 9m28s 4. memcached memcached local latest:ef7d0bf up for 4d11h20m 5. mysql mysql local latest:10ff53f up for 4d11h20m 6. rabbitmq rabbitmq local latest:bed5aec up for 4d11h20m 7. thumbor thumbor local latest:c27d00f up for 4d11h20m 8. consultprofile consultprofile local master:9027f9e up for 4d11h20m 9. platform platform local develop:aebab5b up for 4d11h20m
40. apps can easily bind together (provided they listen envvars) ╭─me@v581 /tmp ╰─$ docker run -t -i --rm --link hbase:hbase quay. io/viadeo/viadeo-consultprofile /bin/bash root@7dd746618873:/app# env | grep HBASE HBASE_ENV_HBASE_HBASE_MASTER_UI_INTERNAL_PORT=60010 HBASE_PORT_2181_TCP_PROTO=tcp HBASE_PORT_60020_TCP_PORT=60020 HBASE_PORT_60010_TCP_PROTO=tcp HBASE_PORT_9090_TCP_PORT=9090 HBASE_PORT_60020_TCP=tcp://172.17.0.41:60020 HBASE_PORT=tcp://172.17.0.41:2181 HBASE_ENV_HBASE_INSTANCES=hbase
41. peaceful integration with our CI (CircleCI) container fast startup reduces build time machine: services: - docker dependencies: post: # stop CircleCI services - sudo /etc/init.d/rabbitmq-server stop # start Docker containers - docker run -d -p 9200:9200 -p 9300:9300 quay.io/viadeo/docker-elasticsearch:latest - docker run -d -p 9201:9200 -p 9301:9300 quay.io/viadeo/docker-elasticsearch:0.20.6 - docker run -d -p 5672:5672 -p 15672:15672 quay.io/viadeo/docker-rabbitmq:latest
42. easy to debug at runtime since we can tail logs of each container (provided they write logs to STDOUT) ╭─me@v581 /tmp ╰─$ docker logs -f platform [main] INFO c.v.e.o.e.plugins - [Dragon Lord] loaded [], sites [] [main] INFO o.r.Reflections - Reflections took 9 ms to scan 1 urls, producing 1 keys and 1 values [main] WARN c.v.p.r.b.DescriptorProvider - 10 resources are marked as expired (@legacy annotation) [main] INFO c.v.p.r.s.HttpConfiguration - Exposing 229 query handlers [main] INFO c.v.p.r.s.HttpConfiguration - Exposing 114 event listeners [main] INFO c.v.p.r.s.HttpConfiguration - Exposing 185 command handlers [main] INFO c.v.p.r.s.HttpConfiguration - Documentation loaded in 377 ms [main] INFO o.e.j.s.Server - jetty-8.1.11.v20130520 Feb 20, 2015 10:05:30 AM
43. we can pop trashable containers to make tests ╭─me@v581 /tmp ╰─$ docker run -t -i --rm -p 2181:2181 quay.io/viadeo/docker- hbase /bin/bash root@6e747d5e0782:/# supervisord -c /etc/supervisor.conf 2015-02-24 21:02:59,144 CRIT Supervisor running as root (no user in config file) 2015-02-24 21:02:59,146 INFO supervisord started with pid 7 2015-02-24 21:03:00,149 INFO spawned: 'zookeeper-server' with pid 8 2015-02-24 21:03:00,150 INFO spawned: 'hbase_master' with pid 9 2015-02-24 21:03:00,151 INFO spawned: 'hbase-regionserver' with pid 10 2015-02-24 21:03:00,152 INFO spawned: 'hbase_thrift' with pid 11 2015-02-24 21:03:01,175 INFO success: zookeeper-server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
44. methodology for building SaaS apps by Adam Wiggins (Heroku) and friends. declarative setup, clean contract, portability, cloud friendly, continuous deployment, scalable, languages & backing services agnostic The DevOps holy bible APPROVED!
45. auto-initialize our containers with data our homemade containers are dev-friendly multi-service container / one-node cluster deploy Docker containers up to the prod? Mesos in the pipeline, all-cloud ambitions containerize all backends/applications? lots of effort for legacy stuff… dockerization of SaaS dependencies? S3, Dynamo Where we go?
46. Mention supplémentaire Questions? Please ask :)