KubeCon EU 2016: Killing containers to make weather beautiful

The Met Office Informatics Lab includes scientists, developers and designers. We build prototypes exploring new technologies to make environmental data useful. Here we describe a recent project to process multi-dimensional weather data to create a fully interactive 4D browser application. We used long-running containers to serve data and web pages and short-running processes to ingest and compress the data. Forecast data is issued every three hours so our data ingestion goes through regular and predictable bursts (i.e. perfect for autoscaling).

We built a Kubernetes cluster in an AWS group which auto-scales based on load. We used replication controllers to process the data. Every three hours ingestion jobs are added to a queue and the number of ingestion containers are set in proportion to the queue length. Each worker completes exactly one ingestion job from the queue and then exits, at which point Kubernetes creates a new one to process the next message. This has allowed us to remove the lifespan logic from the containers and keep them light, fast and massively scalable. We are now in the process of using this in our production systems.

Sched Link: http://sched.co/6BWQ

Transcript

1. Killing containers to
   make weather
   beautiful
   Jacob Tomlinson
   Met Office Informatics Lab
   @_jacobtomlinson
   

   View Slide

2. Who am I?
   

   View Slide

3. View Slide

4. Visualising
   4D data in a
   web
   browser
   Confidence
   in
   integration
   

   View Slide

5. Visualising 4D data in a
   web browser
   

   View Slide

6. Ingestion
   

   View Slide

7. Supercomputer
   Binary files
   PNG images
   Videos
   The browser
   

   View Slide

8. View Slide

9. View Slide

10. Service types
    

    View Slide

11. Long running daemons
    Short running workers
    

    View Slide

12. Kubernetes
    

    View Slide

13. Mixing job types
    

    View Slide

14. Docker Host
    nginx
    System Capacity
    Docker Host
    nginx
    System Capacity
    Docker Host
    nginx
    System Capacity
    

    View Slide

15. Docker Host
    System Capacity
    Docker Host
    System Capacity
    Docker Host
    System Capacity
    nginx nginx nginx
    

    View Slide

16. Docker Host
    nginx
    System Capacity
    Docker Host
    nginx
    System Capacity
    Docker Host
    nginx
    System Capacity
    

    View Slide

17. Docker Host
    nginx
    System Capacity
    Docker Host
    nginx
    System Capacity
    Docker Host
    nginx
    System Capacity
    worker
    worker
    worker
    worker
    worker
    worker
    worker
    worker
    worker
    worker
    worker
    worker
    

    View Slide

18. Docker Host
    nginx
    System Capacity
    Docker Host
    System Capacity
    Docker Host
    System Capacity
    worker
    nginx nginx
    worker
    worker
    

    View Slide

19. Docker
    nginx
    System
    worker
    Docker
    System
    worker
    nginx
    Docker
    System
    worker
    nginx
    Docker
    System
    worker
    worker
    worker
    worker
    worker
    

    View Slide

20. Docker
    System
    worker
    Docker
    System
    worker
    Docker
    System
    worker
    Docker
    System
    worker
    nginx
    worker
    nginx
    nginx
    nginx
    worker
    worker
    worker
    

    View Slide

21. Persistence
    

    View Slide

22. EBS backed
    persistent volumes
    

    View Slide

23. Docker Host
    mongo
    EBS Volume
    Docker Host Docker Host
    

    View Slide

24. Docker Host
    EBS Volume
    Docker Host Docker Host
    mongo mongo
    

    View Slide

25. Docker Host
    EBS Volume
    Docker Host Docker Host
    mongo
    

    View Slide

26. Scaling by queue
    

    View Slide

27. AWS cost vs scale
    

    View Slide

28. Confidence in
    integration
    

    View Slide

29. Short containers
    Long containers
    Daemon processes which listen to requests
    Asynchronous processes doing repetitive
    workloads
    

    View Slide

30. Short containers
    

    View Slide

31. ∞ jobs
    1 job
    n jobs
    

    View Slide

32. Pros
    Conceptual simplicity
    Confidence
    Cons
    Overhead
    

    View Slide

33. Long containers
    

    View Slide

34. Run forever
    Finite runtime
    

    View Slide

35. Scaling down
    

    View Slide

36. Being confident
    

    View Slide

37. Conclusion
    

    View Slide

38. Questions
    Jacob Tomlinson @_jacobtomlinson
    

    View Slide