Upgrade to Pro — share decks privately, control downloads, hide ads and more …

KubeCon EU 2016: Killing containers to make weather beautiful

KubeCon EU 2016: Killing containers to make weather beautiful

The Met Office Informatics Lab includes scientists, developers and designers. We build prototypes exploring new technologies to make environmental data useful. Here we describe a recent project to process multi-dimensional weather data to create a fully interactive 4D browser application. We used long-running containers to serve data and web pages and short-running processes to ingest and compress the data. Forecast data is issued every three hours so our data ingestion goes through regular and predictable bursts (i.e. perfect for autoscaling).

We built a Kubernetes cluster in an AWS group which auto-scales based on load. We used replication controllers to process the data. Every three hours ingestion jobs are added to a queue and the number of ingestion containers are set in proportion to the queue length. Each worker completes exactly one ingestion job from the queue and then exits, at which point Kubernetes creates a new one to process the next message. This has allowed us to remove the lifespan logic from the containers and keep them light, fast and massively scalable. We are now in the process of using this in our production systems.

Sched Link: http://sched.co/6BWQ

Ca3d0556227d66b3c15be1eadf69473b?s=128

Jacob Tomlinson

March 25, 2016
Tweet

Transcript

  1. Killing containers to make weather beautiful Jacob Tomlinson Met Office

    Informatics Lab @_jacobtomlinson
  2. Who am I?

  3. None
  4. Visualising 4D data in a web browser Confidence in integration

  5. Visualising 4D data in a web browser

  6. Ingestion

  7. Supercomputer Binary files PNG images Videos The browser

  8. None
  9. None
  10. Service types

  11. Long running daemons Short running workers

  12. Kubernetes

  13. Mixing job types

  14. Docker Host nginx System Capacity Docker Host nginx System Capacity

    Docker Host nginx System Capacity
  15. Docker Host System Capacity Docker Host System Capacity Docker Host

    System Capacity nginx nginx nginx
  16. Docker Host nginx System Capacity Docker Host nginx System Capacity

    Docker Host nginx System Capacity
  17. Docker Host nginx System Capacity Docker Host nginx System Capacity

    Docker Host nginx System Capacity worker worker worker worker worker worker worker worker worker worker worker worker
  18. Docker Host nginx System Capacity Docker Host System Capacity Docker

    Host System Capacity worker nginx nginx worker worker
  19. Docker nginx System worker Docker System worker nginx Docker System

    worker nginx Docker System worker worker worker worker worker
  20. Docker System worker Docker System worker Docker System worker Docker

    System worker nginx worker nginx nginx nginx worker worker worker
  21. Persistence

  22. EBS backed persistent volumes

  23. Docker Host mongo EBS Volume Docker Host Docker Host

  24. Docker Host EBS Volume Docker Host Docker Host mongo mongo

  25. Docker Host EBS Volume Docker Host Docker Host mongo

  26. Scaling by queue

  27. AWS cost vs scale

  28. Confidence in integration

  29. Short containers Long containers Daemon processes which listen to requests

    Asynchronous processes doing repetitive workloads
  30. Short containers

  31. ∞ jobs 1 job n jobs

  32. Pros Conceptual simplicity Confidence Cons Overhead

  33. Long containers

  34. Run forever Finite runtime

  35. Scaling down

  36. Being confident

  37. Conclusion

  38. Questions Jacob Tomlinson @_jacobtomlinson