DevOps meetup @Viadeo : Proxify everything

At Viadeo, we tend to have internal meetups to share our experiences, needs and vision on how and where things should go.

This talk was about local app proxies in order to abstract complexity and contention on applications configuration files.

Transcript

1. #4 20140307
2. Proxify everything ! There is no place like 127.0.0.1
3. Plan 1 - DevOps movement is fantastic ! 2 - Viadeo-apps configuration files 3 - Proposal 4 - Future - Dynamic configuration updates
4. 1 - DevOps movement is fantastic ! a - The dream b - The doubt c - The situation
5. 1.a - The dream • Devs working with Ops • Ops working with Devs, • Everything in true peace and happiness. ◦ Everybody listen to each other, ◦ Repos are forked, ◦ PRs are raining, ◦ We are agile and birds are singing ! • So far from the begining and the BOFH
6. 1.a - The dream Seriously ? • @ Viadeo, we believed in it, we were enthusiasm, we tried ... • But we did not succeed yet. • As a great man once said : "La route est droite mais la pente est forte !"
8. 1.b - The doubt Questions: What do you think about the situation we are in today ? - Are you satisfied ? - Did it go better ? - Do we have less issues ? - Are we faster ? - Did we break these silos and are we working together ? - What do you feel about today / 2 years ago ? (Seriously no kidding, be true)
9. 1.C - The situation What did we reached ? @Ops point of view, • We can build "systems" in a reproductible and consistent way ! And we can almost do it fast (if we have machines available...) => We have almost removed the "unknown" part of the system => We have improved our trust in the constency of multiple servers, and accross environments*
10. What did we reached ? @Ops point of view, • We tried to package software and make CI pipelines => Safer + More consistent 1.C - The situation
11. 1.C - The situation What did we reached ? @Ops point of view, • We tried to package software and make CI pipelines => Safer + More consistent • But from devs point of view: => Packaging is hard => Packaging /deployment is slow => Packaging brings contraints
12. 1.C - The situation What about the Viadeo’s apps configuration files ?
13. 2 - viadeo-apps configuration files a - Collaboration b - The “prod” c - The challenges
14. 2.a - Collaboration : Still a big pain in the ass Viadeo’s App configuration files managed by Puppet • For “security reason”, Ops did not open their Puppet stuff yet • Ops do not merge the PR fast enougth • We do not know when it will be deployed and if there won't be side effects => We need to be faster => We need to improve QA and tests => BIG Ops internal project (Sorry guys we have to work on these issues too ...)
15. 2.a - Collaboration : Still a big pain in the ass Viadeo Apps conf files are “complex” • Viadeo Apps conf files are different ... how to handle common "values" ◦ Json, Yaml, properties, ini, other ◦ Language specific, ◦ “Key” name different ◦ … • We did not succeeed to SPLIT Infra / Apps specific data YET
16. 2.b - Development / "Production pipeline" The perpetual question… • “Production pipeline”, aka Demo, Preprod, Prod, is handled by Ops and Puppet • Devs like to work on localhost => Vagrant ? “Yeah ... sounds great but it scares me so, maybe but we are not here yet”
17. 2.C - The challenges • How to abstract the infra layer ? • Without having the congestion point at the config file ownership / update velocity ? Some projects have embedded again the conffiles into the project’s jars … :(
18. 3 - The proposal a - Proposition b - Caveats c - Advantages
19. 3.a - Proposition Inspirations : • Twemproxy usage @viadeo • AirBNB : http://nerds.airbnb.com/smartstack-service-discovery-cloud/ + Synapse and local Haproxy instances => standalone local client proxy
20. • For every services, the app is configured to have only 1 server in its server list. • So every operations are send to 127.0.0.1:<servicePort> -> It is often configured that way on Devs workstation -> Ops can provide and guarantee this part easily and consistently thanks to cfgmgnt tools 3.a - Proposition
21. 3.a - Proposition • Memcached / Redis : Local Twemproxy • MySQL : Local ProxySQL / MaxScale • HTTP Services : Local Haproxy with DNS caching, queuing, logging • DNS : Local dnsmasq resolver with DNS query routing • Log collection : Flume Local agent / Logstash • ElasticSeach : Local ES Node without data • ...
22. 3.b - Caveats • It is an additional service to monitor on the “App Box” • It is an additional service in the query processing chain : ◦ Latency, performances issue, debugging step • ...
23. 3.c - Advantages • Infra Abstraction : ◦ Always 127.0.0.1, dev or production (docker make sense then :) ) • Complexity : It's Ops friendly ◦ Easily updatable / automated thanks to cfgmgnt tools => Monitoring is automated => Allow maintenance operations
24. 3.c - Advantages • Performances : May even enhance things: ◦ Multiplexing connections, ◦ Providing queueing / caching facilities to reduce / enhanced the performance issues • Abstract infrastructure even more ◦ ProxySQL can do the query routing that will “solve” our “sharding” model issue that we only faced in production
25. 3.c - Advantages • Troubleshooting : ◦ The query processing chain may even not be extended ◦ Rather than having 1 (or 2) central LB / Proxy called by evey app servers with 2 links network latency… => You have 1 LB / Proxy per app server
26. 4 - Dynamic configuration update
27. 4 - Dynamic configuration update
28. 4 - Dynamic configuration update
29. 4 - Dynamic configuration update
30. 4 - Dynamic configuration update
31. 4 - Dynamic configuration update
32. 4 - Dynamic configuration update
33. 4 - Dynamic configuration update https://speakerdeck.com/philips/etcd-at-gosf
34. Questions ? The end !
35. Links • MySQL ◦ https://github.com/renecannao/proxysql ◦ http://www.skysql.com/downloads/maxscale-mariadb-mysql • Etcd ◦ Puppet Hiera Backend : https://github.com/garethr/hiera-etcd ◦ Etcd and Environment variables : https://github.com/mattn/etcdenv ◦ Daemon listening to keys and updating configuration files in “real- time”, event based : https://github.com/kelseyhightower/confd