Paris container Day 2017 - Take a BlaBlaCar to the stars with rkt - Simon Lallemand - BlaBlaCar

Transcript

1. Take a BlaBlaCar to the stars with rkt ! Simon Lallemand System Engineer @slallema
2. New infrastructure ecosystem Agenda Key principles 100% rkt powered carpooling Directions for driving change BlaBlaCar Facts, Figures & History
3. Founded in 2006 21 million mobile app downloaded (iPhone + Android) Facts and figures 12 million travelers per quarter 1 million tonnes less CO2 in the past year 40 million members Currently in 22 countries France, Spain, UK, Italy, Poland, Hungary, Croatia, Serbia, Romania, Germany, Belgium, India, Mexico, The Netherlands, Luxembourg, Portugal, Ukraine, Czech Republic, Slovakia, Russia, Brazil and Turkey. CO2
4. Our server growth The evolution of the infrastructure from the begining to now: 2006 2008 2010 2012 2013 2014 2015 2016 Web hosting Dedicated servers Baremetal 1 rack Baremetal 3 racks Baremetal 5 racks Baremetal 8 racks Baremetal 14 racks 150 servers 2 DC Baremetal 17 racks 300 servers 3 DC
5. ~300 bare-metal servers ~400 container images 4000+ running containers
6. Tech evolution The major tech changes in our infrastructure : Virtu 2012 Chef 2013 Foreman 2014 Hardware uniformization 2015 Containers 2015 2016
7. Key principles Leading the industrialization at BlaBlaCar
8. Metal is invisible Decouple hardware management and what runs on it
9. Route everything Keep the network simple and scalable
10. Remove snowflakes Be service oriented
11. New Infrastructure Ecosystem 100% rkt powered carpooling
12. Containers For everything !
13. CoreOS Container Linux On 100% of our new servers
14. rkt as container runtime
15. New Infrastructure Ecosystem The tools
16. dgr Container build and runtime tool Harmonize the way we build Quick build One way of doing things Easy to understand for newcomers As little code replication as possible Templating at container start A good integration with rkt github.com/blablacar/dgr
17. dgr : build directory of an ACI ├── aci-manifest.yml ├── attributes │ └── redis.yml ├── runlevels │ └── build │ └── install.sh └── templates └── etc └── redis └── redis.conf.tmpl A standardized structure Ease maintenance and teamwork Inspired by config management Separation of templates, attributes and scripts.
18. dgr : ACI manifest name: redis:0.1 aci: app: exec: - /usr/bin/redis-server - /etc/redis/redis.conf Simpler manifest format Fill only what is important YAML <3 One process per ACI Composition is done only with POD
19. dgr : ACI manifest name: redis:0.1 aci: app: exec: - /usr/bin/redis-server - /etc/redis/redis.conf dependencies: - debian:8.6 Use of dependencies Composition Lighter images
20. dgr : runlevel build ├── aci-manifest.yml ├── attributes │ └── redis.yml ├── runlevels │ └── build │ └── install.sh ├── templates │ └── etc │ └── redis │ └── redis.conf.tmpl └── tests Scripts executed in the container Build from inside the container with all the dependencies
21. dgr : runlevel build #!/bin/bash apt-get install -y redis-server Install packages and stuff For debian based containers you can install packages using apt-get for instance.
22. dgr : runlevel builder name: example.com/aci-redis-dicator:1 builder: dependencies: - example.org/aci-go - example.org/aci-git aci: app: exec: - /bin/dictator Build from outside of the container Dedicated builder image with go & git
23. dgr : runlevel builder #!/dgr/bin/busybox sh git clone \ https://github.com/blablacar/redis-dictator cd redis-dictator go build cp dist/dictator ${ROOTFS}/bin Builder script Clone, build and copy binary to target ACI Only /bin/dictator in final ACI
24. dgr : runlevel builder name: example.org/gentoo-redis:1 builder: dependencies: - example.org/gentoo-stage4 aci: dependencies: - example.org/base app: exec: [ /usr/bin/redis-server ] Other example Builder with gentoo’s stage4
25. dgr : runlevel builder #!/dgr/bin/busybox sh emerge -v redis-server Install packages from outside Use emerge to install your app and its dependencies. Redis with dependencies only in the final ACI
26. dgr : templates & attributes ├── aci-manifest.yml ├── attributes │ └── redis.yml ├── runlevels │ └── build │ └── install.sh └── templates └── etc └── redis └── redis.conf.tmpl Render configuration files Templates stored in the aci Default attributes stored in the aci Overridable when used as dependencies Overridable by environment var
27. dgr : templates & attributes # templates/etc/redis.conf.tmpl daemonize no port {{ .redis.port }} … # attributes/redis.yml default: redis: port: 6379
28. dgr : runlevel prestart ├── aci-manifest.yml ├── attributes │ └── redis.yml ├── runlevels │ └── prestart-late │ └── 10-init-db.sh └── templates └── etc └── redis └── redis.conf.tmpl Initialize container prestart-early and prestart-late scripts Before and after templating Initialize environment before exec
29. dgr : testing ├── aci-manifest.yml ├── attributes ├── runlevels ├── templates └── tests └── wait.sh └── my_cool_tests.bats Testing Bats as default tester wait.sh Wait for service to be ready
30. dgr : testing #!/dgr/bin/bats -x @test "Redis should be running" { run bash -c "ps -aux | grep redis-server" [ "$status" -eq 0 ] echo ${lines[0]} [[ "${lines[0]}" =~ "/usr/bin/redis-server" ]] } @test "Redis should listen on port: 6379" { run bash -c "netstat -peanut | grep redis-server" [ "$status" -eq 0 ] [[ "${lines[0]}" =~ ":6379" ]] [[ "${lines[0]}" =~ "redis-server" ]] }
31. dgr : subcommands ~ # dgr init ~ # dgr try ~ # dgr build ~ # dgr test ~ # dgr install ~ # dgr push github.com/blablacar/dgr
32. ggn Manage services in fleet clusters Pods Services and Environments Different services that use the same pods Environments (1 per DC for prod + dev…) Abstraction of fleet commands github.com/blablacar/ggn INSERT LOGO HERE
33. ggn ├── prod-dc1 │ └── attributes │ └── services │ └── redis-quota │ │ └── attributes │ │ └── service-manifest.yml │ │ └── unit.tmpl │ └── redis-rating ├── preprod │ ... Single directory tree Describes the services The environments Versioned in git
34. ggn : service manifest containers: - blablacar.com/aci-redis:3 - blablacar.com/aci-redis-exporter:1 nodes: - hostname: redis-quota1 - hostname: redis-quota2 - hostname: redis-quota3 Define the service ACI images and versions combined to make a pod Nodes composing the service
35. ggn : attributes # attributes/redis.yml override: redis: maxmemory: 4GB TEMPLATER_OVERRIDE={“redis”:{“max memory”: “4GB”}…} Override attributes Inject run context attributes as environment var
36. ggn : unit.tmpl [Unit] Description=Redis POD {{. [Service] KillMode=mixed Restart=always ExecStart=/usr/bin/rkt run \ --set-env=TEMPLATER_OVERRIDE='{{.jsonAttrs}}' {{.acis}}
37. ggn : run a POD $ ggn prod-dc1 redis-quota update $ ggn prod-dc1 redis-quota1 journal -f github.com/blablacar/ggn
38. Service discovery with nerve & synapse AirBnB’s smartstack Dynamic topology of services Scalability requires service discovery We started with smartstack of AirBnB Since then we improved it and rewrote it in go github.com/blablacar/go-nerve github.com/blablacar/go-synapse
39. /database/node1 go-nerve does health checks and reports to zookeeper in service keys HAProxy node1 Applications hit their local haproxy to access backends Service Discovery /database go-synapse watches zookeeper service keys and reloads haproxy if changes are detected go-nerve Zookeeper go-synapse
40. bare-metal servers 1 type of hardware 3 disk profiles fleet machines CoreOS fleet etcd “Distributed init system” Hardware Container Registry ggn dgr Service Codebase rkt PODs build run store host create Our infrastructure ecosystem nerve mysqld monitoring mysql-main_1 nerve php monitoring nginx synapse front_1 synapse nerve zookeeper Service Discovery
41. Work In Progress ( We’re hiring )
42. @slallema @BlaBlaCarTech
43. Thanks! @BlaBlaCarTech BlaBlaTech.com