Kubernetes, Ingress and Traefik Usage at CERN Ricardo Rocha CERN Cloud Team 

About Computing Engineer in the CERN cloud team Focusing on containers, kubernetes and networking Accelerators and ML Previous work in storage and the WLCG (worldwide LHC computing grid) @ahcorporto [email protected] 

Founded in 1954 What is 96% of the universe made of? Fundamental Science Why isn’t there anti-matter in the universe? What was the state of matter just after the Big Bang? 

No content

No content

No content

No content

8 

No content

10 

No content

No content

~70 PB/year 700 000 Cores ~400 000 Jobs ~30 GiB/s 200+ Sites 

Computing at CERN Increased numbers, increased automation 1970s 2007 

Computing at CERN Increased numbers, increased automation 1970 2007 

Computing at CERN Increased numbers, increased automation 1970 2007 

Computing at CERN Increased numbers, increased automation 1970 2007 

Provisioning Deployment Update Physical Infrastructure Days or Weeks Minutes or Hours Minutes or Hours Utilization Poor Maintenance Highly Intrusive Cloud API Virtualization Minutes Minutes or Hours Minutes or Hours Good Potentially Less Intrusive Containers Seconds Seconds Seconds Very Good Less Intrusive 

Simplified Infrastructure Monitoring, Lifecycle, Alarms Simplified Deployment Uniform API, Replication, Load Balancing Periodic Load Spikes International Conferences, Reprocessing Campaigns 

Use Cases 

1 PB / sec < 10 GB / sec Typically split into Hardware and Software Filters ( this might change too ) 40 million particle interactions / second ~3000 multi-core nodes ~30.000 applications to supervise Critical system, sustained failure means data loss Can it be improved for Run 4? Study 2017, Mattia Cadeddu, Giuseppe Avolio Kubernetes 1.5.x A new evaluation phase to be tried this year ATLAS Event Filter 

How to efficiently distribute experiment software? CernVM-FS (cvmfs): a read-only, hierarchical filesystem In production for several years, battle tested, solved problem Now with containers? Can they carry all required software? > 200 sites in our computing grid ~400 000 concurrent jobs Frequent software releases 100s of GBs 

Docker Images of ~10GB+ Poorly Layered, Frequently Updated Clusters of 100s of nodes Can we have lazy image distributions? And file level granularity? And caches? Containerd Remote Snapshotter https://bit.ly/3bdkLmh 

Simulation is one of our major computing workloads x100 soon as described early Deep Learning for Fast Simulation Can we easily distribute to reduce training time? Sofia Vallecorsa, CERN OpenLab Konstantinos Samaras-Tsakiris 

ATLAS Production System Running a Grid site is not trivial We have > 200 of them Multiple components for Storage and Compute Lots of history in the software Fernando Barreiro Megino Fahui-Lin, Mandy Yang ATLAS Distributed Computing Can a Kubernetes endpoint be a Grid site? 

1st attempt to ramp up. K8s master running on Medium VM Master killed (OOM) on Saturday Test Cluster with 2000 cores Good: Initial results show error rates as any other site Improvements: defaults on the scheduler causing inefficiencies Pack vs Spread Affinity Predicates, Weights Custom Scheduler? 

No content

No content

Cluster as a Service API / CLI to create, update, resize, delete clusters Ability to pass labels for custom clusters (specific features, versions, components) openstack coe cluster template list | kubernetes-1.17.9-2 | | kubernetes-1.18.6-3 | openstack coe cluster create --cluster-template kubernetes-1.18.6-3 \ --node-count 10 --flavor m2.large \ --labels nvidia_gpu_enabled=true \ mytestcluster openstack coe nodegroup create \ --label availability_zone=cern-geneva-a --node-count 3 ... 

Common Stack Fedora Core as base, immutable OS Containerd / runc as container runtime - relying on CRI Kubernetes as the container orchestrator Fluentd for log collection and aggregation Prometheus for monitoring and metric collection 

Ingress: Traefik Traefik has been our default ingress controller from day one Great integration, healthy community and feedback Covered all our initial use cases 

Traefik and Ingress Master Node Node Master update dns role=ingress Node Node Node Network DB myservice.cern.ch apiVersion: extensions/v1beta1 kind: Ingress metadata: name: myservice-ingress annotations: kubernetes.io/ingress.class: traefik spec: rules: - host: myservice.cern.ch http: paths: - path: / backend: serviceName: myservice servicePort: 8080 Sync watch ingress watch/set nodes 

Ingress: Traefik and Simple HTTP The simple Ingress definition covers most of our use cases In most cases SSL termination is good enough kind: Ingress metadata: name: myservice-ingress-tls annotations: kubernetes.io/ingress.class: traefik traefik.ingress.kubernetes.io/frontend-entry-points: https spec: rules: - host: myservice.cern.ch http: paths: - path: / backend: serviceName: myservice servicePort: 8080 

Ingress: Traefik and ACME / Let’s Encrypt Easy, popular solution DNS challenge is not yet an options No API available to update TXT records We rely on the HTTP-01 challenge This requires a firewall opening to get a certificate, not ideal 

Ingress: Traefik and Client Certificate Some services require info from the client certificate Annotations start being larger than the core Ingress resource definition annotations: kubernetes.io/ingress.class: traefik traefik.ingress.kubernetes.io/frontend-entry-points: https traefik.ingress.kubernetes.io/pass-client-tls-cert: | pem: true infos: notafter: true notbefore: true sans: true subject: country: true province: true locality: true organization: true commonname: true serialnumber: true 

Ingress: Other Requirements SSL Passthrough Exposing TCP ports HTTP Header based redirection 

Conclusion + Next Steps Traefik has been very stable in our deployments We need to move Traefik to 2.0 - yes we’re still in 1.x Most used Ingress controller by far - almost 400 clusters using it at CERN Integrate Ingress with our external LBs - using VIP, no DNS Monitor developments of the new Service APIs https://github.com/kubernetes-sigs/service-apis 

Webinars https://clouddocs.web.cern.ch/containers/training.html#webinars 

Questions?