How have we been building a containers-based PaaS these last 5 years?

Talk about building a PaaS, and container scheduling system

Transcript

1. How have we been building a containers-based PaaS these last 5 years? Soulou 26 Juin 2018
2. Léo Unbekandt, Co-founder & CTO Automagic cloud hosting for awesome teams Quick Presentation
3. Part 1 Genesis of a containers- backed PaaS
4. What is Scalingo? Third-party application hosting platform → Images built from code source Database as a Service hosting → Pre-built images available on Docker Hub No Docker knowledge required
5. DBaaS Offer What is Scalingo?
6. Constraints PaaS provider objective ≠ Software service company Infrastructure optimization High level of consolidation Efficient scheduling → Avoid paying unused resources
7. Back to the past (2013) Once upon a time...
8. Containers? Scheduling? “Linux Containers” Jan 2013
9. Tooling timeline Tools Docker 0.1.0 → 2013-03-23 Etcd 0.1.0 → 2013-08-11 Orchestration Swarm 1.0.0 → 2015-11-03 Kubernetes 1.0.6 → 2015-09-11
10. Sept 2013: first prototype Basic Scheduling Container Clustering
11. Growing 2013: 10s 2015: 100s 2018: 1000s
12. Part 2 Containers == Docker? Small hint → NO
13. A new Standard? Containers under the spotlight
14. Ecosystem
15. A (small) swissknife A Little of everything Not efficient everywhere Lock-in → A tool for us: Isolation helper Apps distribution
16. Logging 2013 <-> 2016: “JSON file tailing” (which eventually breaks on logrotate, still today YES.)
17. Logging - demo $ docker run -it --log-opt file-max=1 --log-opt file-size=1m ubuntu:16.04 bash $ for i in $(seq 100000) ; do perl -e "print 'xxxxxxxxxxxxxxxxxxxxxxxxx$i' x 10, \"\n\"" ; done $ docker logs --follow <id> Demo: breaking docker logs
18. Logging Agent TCP + TLS Agent TCP + TLS Message Bus 1.10.0 (2016-02-04) : Syslog driver TCP+TLS
19. Monitoring GET /containers/:id/stats - No configuration possible - Poor performance
20. Monitoring Agent Message Bus Cgroups Namespaces Agent Cgroups Namespaces https://github.com/Scalingo/acadock-monitoring
21. Networking throttling Still searching...
22. Networking throttling $ interface_id=$(echo 'ip link show eth0' | nsenter --target #{pid} --net --pid) $ iface=$(ip link show | grep “^${interface_id}:”) $ tc qdisc replace dev “${iface}” root tbf rate “${limit}”mbit latency 200ms burst “${burst}”MB No hook possible with Docker → Our job
23. Networking Docker should be a tool → No lock-in Goodbye Docker network
24. Software Distribution Docker Registry Image spreading
25. Security Concerns Unprivileged users only 1 container → 1 user Apps are built with 1 layer → 1 tarball Ability to patch base image
26. Part 3 Orchestration aka. Clustering and Scheduling
27. Simple bridges and port exports Networking - Apps App App App App HTTP Requests Bridge
28. Networking - Database Clusters DB DB DB Internal Cluster Network Per cluster private network
29. Networking - SAND https://github.com/Scalingo/sand
30. NFS → Good enough to start Downsides: • SPoF • HA tricky to get • Hard to scale • No blkio cgroup! Storage - Beginnings
31. Storage - NFS HA NFS DB DB DB DB NFS DRBD VIP DB DB DB DB MASTER FAILOVER
32. Software Defined Storage - GlusterFS* - Ceph* - OpenSDS* - OpenEBS* - Rook* - StorageOS - Hedvig - ScaleIO (Dell) - Kasten - Virtuoso - Datacore - Diamanti - Hatchway (VMWare) - Portworx - Quobyte - Datera - Robin Cloud Platform - ... * FOSS
33. Attached disks from SAN ≥ 100 volumes per host LVM magic Storage ie. https://github.com/Scalingo/go-fssync Good cloning tools Reattach on host failure
34. Little Reminder Our goal: optimize servers usage Automate all the things Include business logics Scheduling
35. Stop the world and move things Online vs Offline New container to an existing topology Scheduling
36. Scheduling - Online - Multiplicity of strategies - Memory - CPU - Load average /!\ - Disk IO
37. Scheduling - Online - E. Arzuaga and D. R. Kaeli. Quantifying load imbalance on virtualized enterprise servers. page 235. ACM Press. - D. Ferrari and S. Zhou. An empirical investigation of load indices for load balancing applications. - T. Wood, P. Shenoy, A. Venkataramani, and M. Yousif. Sandpiper : Black-box and gray-box resource management for virtual machines. 53(17) : 2923–2938. Current Strategy → Sandpiper
38. Scheduling - Offline
39. Scheduling - Offline (And listen to the monitoring)
40. Why not existing tech today? ???
41. Business Logics and Support → Control needed on core infrastructure → Technology mastering → Need to patch K8s? → Scheduling strategy (w/ overcommitting) → “Unstable” solution Why not existing tech today?
42. Owned Apps vs Third-Party → Not design for third-party → Billing requirements → Fine grained resource control Why not existing tech today?
43. Soulou Thanks! https://scalingo.com 30-days free trial Try us: PCD2018 (50€ voucher) Questions?