KubeCon EU 2016: Killing containers to make weather beautiful

The Met Office Informatics Lab includes scientists, developers and designers. We build prototypes exploring new technologies to make environmental data useful. Here we describe a recent project to process multi-dimensional weather data to create a fully interactive 4D browser application. We used long-running containers to serve data and web pages and short-running processes to ingest and compress the data. Forecast data is issued every three hours so our data ingestion goes through regular and predictable bursts (i.e. perfect for autoscaling).

We built a Kubernetes cluster in an AWS group which auto-scales based on load. We used replication controllers to process the data. Every three hours ingestion jobs are added to a queue and the number of ingestion containers are set in proportion to the queue length. Each worker completes exactly one ingestion job from the queue and then exits, at which point Kubernetes creates a new one to process the next message. This has allowed us to remove the lifespan logic from the containers and keep them light, fast and massively scalable. We are now in the process of using this in our production systems.

Sched Link: http://sched.co/6BWQ

Transcript

1. Killing containers to make weather beautiful Jacob Tomlinson Met Office Informatics Lab @_jacobtomlinson
2. Who am I?
4. Visualising 4D data in a web browser Confidence in integration
5. Visualising 4D data in a web browser
6. Ingestion
7. Supercomputer Binary files PNG images Videos The browser
10. Service types
11. Long running daemons Short running workers
12. Kubernetes
13. Mixing job types
14. Docker Host nginx System Capacity Docker Host nginx System Capacity Docker Host nginx System Capacity
15. Docker Host System Capacity Docker Host System Capacity Docker Host System Capacity nginx nginx nginx
16. Docker Host nginx System Capacity Docker Host nginx System Capacity Docker Host nginx System Capacity
17. Docker Host nginx System Capacity Docker Host nginx System Capacity Docker Host nginx System Capacity worker worker worker worker worker worker worker worker worker worker worker worker
18. Docker Host nginx System Capacity Docker Host System Capacity Docker Host System Capacity worker nginx nginx worker worker
19. Docker nginx System worker Docker System worker nginx Docker System worker nginx Docker System worker worker worker worker worker
20. Docker System worker Docker System worker Docker System worker Docker System worker nginx worker nginx nginx nginx worker worker worker
21. Persistence
22. EBS backed persistent volumes
23. Docker Host mongo EBS Volume Docker Host Docker Host
24. Docker Host EBS Volume Docker Host Docker Host mongo mongo
25. Docker Host EBS Volume Docker Host Docker Host mongo
26. Scaling by queue
27. AWS cost vs scale
28. Confidence in integration
29. Short containers Long containers Daemon processes which listen to requests Asynchronous processes doing repetitive workloads
30. Short containers
31. ∞ jobs 1 job n jobs
32. Pros Conceptual simplicity Confidence Cons Overhead
33. Long containers
34. Run forever Finite runtime
35. Scaling down
36. Being confident
37. Conclusion
38. Questions Jacob Tomlinson @_jacobtomlinson