STARTS WITH A SIMPLE PRODUCT IT ALWAYS STARTS WITH A SIMPLE PRODUCT You want to change the world by disrupting the job board industry Standard three-tier, self-contained app Does not fall into the usual definition of distributed systems 12 . 1
OF SUCCESS FIRST SIGNS OF SUCCESS Your single server is not sufficient anymore Database gets its own machines, adding new web servers fixes the issue Logging becomes a bit harder You switch to a centralized logging solution 14 . 1
ADDED FEATURES GET ADDED Subscriptions emails Doing it synchronously is impossible Let's add a worker (and thus a queueing mechanism) You start switching from pets to cattle 16 . 1
RUNS OUT SEED MONEY RUNS OUT Let's try other monetization techniques Freemium model with analytics Where do I run these batch jobs? You partner with another company to exchange data They have this weird legacy system and the only client lib is in PHP :-( 18 . 1
PRODUCT GROWS, SO DOES AS THE PRODUCT GROWS, SO DOES INFRASTRUCTURE INFRASTRUCTURE You're now at 3 jenkins workers You had to split metrics and monitoring on separate machines You introduce a command and control solution to perform your regular operations It's time to use puppet (You're really starting to feel like an ops person now) 20 . 1
UTILIZATION IS LOW BUT RESOURCE UTILIZATION IS LOW Most of it is articifial (agents on every nodes) But you still have peak induced regular contention 22 . 1
SERVICES OR COMPONENTS IS HARD ADDING NEW SERVICES OR COMPONENTS IS HARD Should your most active git repository really be the puppet one? You constantly have to make allocation decisions 23 . 1
IS HARD HANDLING FAILURE IS HARD Your monitoring system tells you when something breaks You have to recreate machines manually, update configuration all over the place 24 . 1
ADDITIONAL PROMISES A good substrate for creating resources on- demand Lingua-franca for infrastructure concepts Initial learning-curve Most likely smaller than ad-hoc solutions Eating our own dog food 32 . 1
Building once (and in chroots) ensures clean packages Reproducible builds make wide changes easier We need staging deploys and production deploys to be identical 36 . 1
PACKAGING AFTER KUBERNETES: PACKAGING Building is faster, easier, and gives developers more autonomy Docker registries forced us to move from enforcement to convention 49 . 1
CONFIGURATION AFTER KUBERNETES: CONFIGURATION The split across environment services, variables, configmaps, and secrets makes separation easy Greatly reduces the need for config management Configuration can be colocated with the so ware Removes the code, build, and configuration-management impedance mismatch Kept things simple No helm 50 . 1
ON-DEMAND AFTER KUBERNETES: ON-DEMAND RESOURCES RESOURCES CRDs provide great integration High cardinality or complex queries can be tedious to work with 51 . 1
SECURITY AFTER KUBERNETES: SECURITY RBAC policies are powerful but tedious to write (and error prone) Certificate management leaves a lot to be improved 52 . 1
EXTERNAL SERVICES WHAT ABOUT EXTERNAL SERVICES By default NodePort services run on all worker nodes Traffic is source-nat'd to the destination Losing source IP information is unviable for most use- cases Performance impact 61 . 1
EXOSCALE NETWORKING AT EXOSCALE A layer3 all-the-way design A BGP first design VM Public IPs advertised by BGP from hypervisors Private network VXLAN membership advertised through BGP- eVPN Best performance is on the public interfaces 62 . 1
EXTERNAL SERVICES Needs additional development We went for IPIP load-balancing Needs node-local decapsulation Makes for a hacky setup eBPF/XDP to the rescue! 65 . 1
EXOSCALE TODAY KUBERNETES AT EXOSCALE TODAY Most production critical services Some things we are in no hurry to containerize :-) Basis for as-a-service offerings Private network management For customers exo lab kube Cluster API OpenShi On-demand cluster in the Exoscale API 66 . 1