Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Pathfinder - Building a Container Platform in R...

Pathfinder - Building a Container Platform in Ruby Ecosystem

This session will discuss about an attempt to build container platform in ruby/mruby ecosystem, the current situation and lesson-learned that we can discern to improve it further.

Further on, for those whom are unfamiliar, this session will also touch a bit about container platform/orchestrator and the generic architecture behind it. So that as a developer, we understand the abstraction that it provides.

*This deck was presented in RubyKaigi 2019*

Avatar for Giovanni Sakti

Giovanni Sakti

April 18, 2019
Tweet

More Decks by Giovanni Sakti

Other Decks in Programming

Transcript

  1. This session will discuss about an attempt to build container

    platform in ruby/mruby ecosystem, the current situation and lesson-learned that we can discern to improve it further. Further on, for those whom are unfamiliar, this session will also touch a bit about container platform/orchestrator and the generic architecture behind it. So that as a developer, we understand the abstraction that it provides.
  2. a software that allows us to manage many computers as

    if they were a single big computer and assign jobs into it
  3. Data Storage The Memory Pathfinder Mono State Server & UI

    The Conscious Pathfinder Scheduler The Brain Container #1 Container #2 Container #3 Pathfinder Agent The Sidecar Container Runtime Pathfinder CLI
  4. Data Storage The Memory Pathfinder Mono State Server & UI

    The Conscious Pathfinder Scheduler The Brain Container #1 Container #2 Container #3 Pathfinder Agent The Sidecar Container Runtime Pathfinder CLI
  5. Data Storage The Memory Pathfinder Mono State Server & UI

    The Conscious Pathfinder Scheduler The Brain Container #1 Container #2 Container #3 Pathfinder Agent The Sidecar Container Runtime Pathfinder CLI
  6. Data Storage Pathfinder Mono State Server & UI Pathfinder Scheduler

    Container #1 Container #2 Container #3 Pathfinder Agent Container Runtime (LXD, Docker, Containerd, etc) Pathfinder CLI The Memory The Conscious The Brain The Sidecar
  7. Data Storage The Memory Pathfinder Mono State Server & UI

    The Conscious Pathfinder Agent The Sidecar Container Runtime Self-registration
  8. Data Storage The Memory Pathfinder Mono State Server & UI

    The Conscious Pathfinder Scheduler The Brain
  9. Data Storage The Memory Pathfinder Mono State Server & UI

    The Conscious Pathfinder Scheduler The Brain
  10. Data Storage The Memory Pathfinder Mono State Server & UI

    The Conscious Pathfinder Agent The Sidecar Container Runtime
  11. Pathfinder in Production 1. 50+ worker nodes 2. ~1000 containers

    3. Already live for 6 months+ 4. Serving our logging infrastructures (for now)
  12. Data Storage The Memory Pathfinder Mono State Server & UI

    The Conscious Pathfinder Scheduler The Brain Container #1 Container #2 Container #3 Pathfinder Agent The Sidecar Container Runtime Pathfinder CLI
  13. Data Storage The Memory Pathfinder Mono State Server & UI

    The Conscious Pathfinder Scheduler The Brain Container #1 Container #2 Container #3 Pathfinder Agent The Sidecar Container Runtime Pathfinder CLI
  14. What the agent does Interact with the state server •

    Do self-registration • Make queries
  15. Use case: Interact with the state server • Do self-registration

    • Make queries Need: a way to communicate with REST API server via HTTP/HTTPS.
  16. Use case: Interact with local container daemon • Make queries

    • Execute changes Need: • a way to communicate with REST API server via UNIX socket • Communicate by following LXC daemon (LXD) API documentation
  17. Use case: Collect and send metrics Need: It needs to

    gather system metrics (CPU & RAM utilization at minimum)
  18. Data Storage The Memory Pathfinder Mono State Server & UI

    The Conscious Pathfinder Scheduler The Brain Container #1 Container #2 Container #3 Pathfinder Agent The Sidecar Container Runtime Pathfinder CLI
  19. Now

  20. Data Storage The Memory Pathfinder Mono State Server & UI

    The Conscious Pathfinder Scheduler The Brain Container #1 Container #2 Container #3 Pathfinder Agent The Sidecar Container Runtime Pathfinder CLI
  21. List of Projects - Reusable libraries • giosakti/mruby-lxd Libraries for

    interacting with LXC Daemon (LXD) • pathfinder-cm/mruby-pathfinder-client Libraries for interacting with Pathfinder state server • giosakti/mruby-cmdr Mini frameworks for structuring CLI apps
  22. List of Projects – Rewritten Components • pathfinder-cm/pathfinder-agent-mruby Agent that

    will be installed in the worker nodes • pathfinder-cm/pfi-mruby CLI for interacting with Pathfinder state server
  23. Documentation of mruby is a bit lacking One area of

    improvement that we can all help