Pathfinder - Building a Container Platform in Ruby Ecosystem

This session will discuss about an attempt to build container platform in ruby/mruby ecosystem, the current situation and lesson-learned that we can discern to improve it further.

Further on, for those whom are unfamiliar, this session will also touch a bit about container platform/orchestrator and the generic architecture behind it. So that as a developer, we understand the abstraction that it provides.

*This deck was presented in RubyKaigi 2019*

Transcript

1. Building a Container Platform
   in ruby ecosystem
   Rubykaigi 2019 @giosakti
   

   View Slide

2. Hello
   

   View Slide

3. 名前わギオ
   

   View Slide

4. 名前わギオ
   My name is Gio
   

   View Slide

5. Was the Kanji correct?
   Sorry I used google translate
   

   View Slide

6. I come from Jakarta
   

   View Slide

7. View Slide

8. I work at GO-JEK
   

   View Slide

9. I work at GO-JEK
   Anybody knows GO-JEK or ゴ-ジェック?
   

   View Slide

10. View Slide

11. View Slide

12. Super
    App.
    

    View Slide

13. And we ❤ Ruby
    

    View Slide

14. Ruby is one of 5 most
    used languages
    in the company
    

    View Slide

15. I also organized ruby
    meetups in Jakarta
    

    View Slide

16. View Slide

17. ruby.id/conf
    

    View Slide

18. Moving on…
    

    View Slide

19. Building a Container Platform
    in ruby ecosystem
    

    View Slide

20. This session will discuss about an attempt to build
    container platform in ruby/mruby ecosystem, the
    current situation and lesson-learned that we can
    discern to improve it further.
    Further on, for those whom are unfamiliar, this session
    will also touch a bit about container
    platform/orchestrator and the generic architecture
    behind it. So that as a developer, we understand the
    abstraction that it provides.
    

    View Slide

21. Background
    

    View Slide

22. This started early last
    year...
    

    View Slide

23. Converging to
    centralized platform
    

    View Slide

24. Principles & Considerations
    Hybrid
    Infrastructures
    Explore
    Containers
    Seamless
    Transition
    Open Source
    

    View Slide

25. One of projects that I
    involved was Logging
    

    View Slide

26. github.com/BaritoLog
    

    View Slide

27. We decided to use
    LXC to host Barito
    components
    

    View Slide

28. Why LXC?
    1. Drop-in VM replacement
    2. We have cookbooks for all the components
    

    View Slide

29. But, we need a system
    to manage those LXCs
    

    View Slide

30. View Slide

31. github.com/pathfinder-cm
    

    View Slide

32. A container platform.
    

    View Slide

33. Written in Ruby and
    Golang
    

    View Slide

34. But, what exactly is a
    container platform?
    

    View Slide

35. But, what exactly is a
    container platform?
    (sometimes it is also called orchestrator or manager)
    

    View Slide

36. I found this online
    

    View Slide

37. http://splasho.com/upgoer5/
    

    View Slide

38. View Slide

39. Neat.
    

    View Slide

40. a software that allows us to
    manage many computers as if
    they were a single big computer
    and assign jobs into it
    

    View Slide

41. What it means is that
    we…
    

    View Slide

42. Abstract away CPU & RAM
    

    View Slide

43. Abstract away Storage
    

    View Slide

44. Abstract away Network
    

    View Slide

45. The Architecture
    

    View Slide

46. Data Storage
    The Memory
    Pathfinder Mono
    State Server & UI
    The Conscious
    Pathfinder
    Scheduler
    The Brain
    Container #1
    Container #2
    Container #3
    Pathfinder
    Agent
    The Sidecar
    Container
    Runtime
    Pathfinder CLI
    

    View Slide

47. Data Storage
    The Memory
    Pathfinder Mono
    State Server & UI
    The Conscious
    Pathfinder
    Scheduler
    The Brain
    Container #1
    Container #2
    Container #3
    Pathfinder
    Agent
    The Sidecar
    Container
    Runtime
    Pathfinder CLI
    

    View Slide

48. Data Storage
    The Memory
    Pathfinder Mono
    State Server & UI
    The Conscious
    Pathfinder
    Scheduler
    The Brain
    Container #1
    Container #2
    Container #3
    Pathfinder
    Agent
    The Sidecar
    Container
    Runtime
    Pathfinder CLI
    

    View Slide

49. Data Storage
    Pathfinder Mono
    State Server & UI
    Pathfinder
    Scheduler
    Container #1
    Container #2
    Container #3
    Pathfinder
    Agent
    Container
    Runtime
    (LXD, Docker,
    Containerd,
    etc)
    Pathfinder CLI
    The Memory
    The Conscious
    The Brain
    The Sidecar
    

    View Slide

50. Pathfinder in Action
    

    View Slide

51. Data Storage
    The Memory
    Pathfinder Mono
    State Server & UI
    The Conscious
    Pathfinder
    Agent
    The Sidecar
    Container
    Runtime
    Self-registration
    

    View Slide

52. Data Storage
    The Memory
    Pathfinder Mono
    State Server & UI
    The Conscious
    Pathfinder CLI
    

    View Slide

53. Data Storage
    The Memory
    Pathfinder Mono
    State Server & UI
    The Conscious
    Pathfinder CLI
    

    View Slide

54. Data Storage
    The Memory
    Pathfinder Mono
    State Server & UI
    The Conscious
    Pathfinder CLI
    

    View Slide

55. Data Storage
    The Memory
    Pathfinder Mono
    State Server & UI
    The Conscious
    Pathfinder
    Scheduler
    The Brain
    

    View Slide

56. Data Storage
    The Memory
    Pathfinder Mono
    State Server & UI
    The Conscious
    Pathfinder
    Scheduler
    The Brain
    

    View Slide

57. Data Storage
    The Memory
    Pathfinder Mono
    State Server & UI
    The Conscious
    Pathfinder
    Agent
    The Sidecar
    Container
    Runtime
    

    View Slide

58. View Slide

59. Pathfinder in Production
    1. 50+ worker nodes
    2. ~1000 containers
    3. Already live for 6 months+
    4. Serving our logging infrastructures (for now)
    

    View Slide

60. Pathfinder in Production
    Roll-out process is still ongoing.
    Expect at least 3-4x the current traffic.
    

    View Slide

61. Pathfinder in Ruby
    

    View Slide

62. Data Storage
    The Memory
    Pathfinder Mono
    State Server & UI
    The Conscious
    Pathfinder
    Scheduler
    The Brain
    Container #1
    Container #2
    Container #3
    Pathfinder
    Agent
    The Sidecar
    Container
    Runtime
    Pathfinder CLI
    

    View Slide

63. Mruby is particularly interesting
    1. Easily create executable
    2. Low footprint
    3. Uniform language
    

    View Slide

64. Possible challenges
    Ecosystem?
    

    View Slide

65. Let's Find Out
    

    View Slide

66. Writing Pathfinder Agent
    in mruby
    

    View Slide

67. Data Storage
    The Memory
    Pathfinder Mono
    State Server & UI
    The Conscious
    Pathfinder
    Scheduler
    The Brain
    Container #1
    Container #2
    Container #3
    Pathfinder
    Agent
    The Sidecar
    Container
    Runtime
    Pathfinder CLI
    

    View Slide

68. What the agent does
    Interact with the state server
    • Do self-registration
    • Make queries
    

    View Slide

69. What the agent does
    Interact with local container daemon
    • Make queries
    • Execute changes
    

    View Slide

70. What the agent does
    Collect and send metrics
    

    View Slide

71. Let’s discuss the need of
    each use-case
    

    View Slide

72. Use case:
    Interact with the state server
    • Do self-registration
    • Make queries
    Need:
    a way to communicate with REST API server via
    HTTP/HTTPS.
    

    View Slide

73. Used:
    github.com/matsumotory/mruby-
    simplehttp
    

    View Slide

74. Use case:
    Interact with local container daemon
    • Make queries
    • Execute changes
    Need:
    • a way to communicate with REST API server
    via UNIX socket
    • Communicate by following LXC daemon
    (LXD) API documentation
    

    View Slide

75. Used:
    github.com/matsumotory/mruby-
    simplehttp
    

    View Slide

76. Extracted:
    github.com/giosakti/mruby-lxd
    

    View Slide

77. Use case:
    Collect and send metrics
    Need:
    It needs to gather system metrics (CPU & RAM
    utilization at minimum)
    

    View Slide

78. Extracted:
    github.com/giosakti/mruby-
    sysinfo
    

    View Slide

79. Use case:
    Build self-contained
    executable
    

    View Slide

80. Used:
    github.com/hone/mruby-cli
    

    View Slide

81. Writing Pathfinder CLI
    in mruby
    

    View Slide

82. Data Storage
    The Memory
    Pathfinder Mono
    State Server & UI
    The Conscious
    Pathfinder
    Scheduler
    The Brain
    Container #1
    Container #2
    Container #3
    Pathfinder
    Agent
    The Sidecar
    Container
    Runtime
    Pathfinder CLI
    

    View Slide

83. What the CLI does
    Allow user to interact with the APIs of state
    server
    

    View Slide

84. The requirements are
    similar to agent
    

    View Slide

85. Extracted:
    pathfinder-cm/pathfinder-mruby-client
    

    View Slide

86. Extracted:
    giosakti/mruby-cmdr
    

    View Slide

87. 2 other components are
    already in ruby
    

    View Slide

88. Pathfinder Mono
    The state server
    

    View Slide

89. Pathfinder Scheduler
    

    View Slide

90. Now
    

    View Slide

91. Data Storage
    The Memory
    Pathfinder Mono
    State Server & UI
    The Conscious
    Pathfinder
    Scheduler
    The Brain
    Container #1
    Container #2
    Container #3
    Pathfinder
    Agent
    The Sidecar
    Container
    Runtime
    Pathfinder CLI
    

    View Slide

92. List of Projects - Reusable libraries
    • giosakti/mruby-lxd
    Libraries for interacting with LXC Daemon (LXD)
    • pathfinder-cm/mruby-pathfinder-client
    Libraries for interacting with Pathfinder state server
    • giosakti/mruby-cmdr
    Mini frameworks for structuring CLI apps
    

    View Slide

93. List of Projects – Rewritten Components
    • pathfinder-cm/pathfinder-agent-mruby
    Agent that will be installed in the worker nodes
    • pathfinder-cm/pfi-mruby
    CLI for interacting with Pathfinder state server
    

    View Slide

94. List of Projects – Existing Components
    • pathfinder-cm/pathfinder-mono
    State server & scheduler on one repo
    

    View Slide

95. Wrap-up
    

    View Slide

96. No Gemfile ☺
    

    View Slide

97. mruby has almost all the
    libraries we need to do
    such projects
    

    View Slide

98. Building the executable for
    mruby application is fairly
    straightforward
    

    View Slide

99. The agent mruby executable
    is smaller than the go one
    Need to investigate further
    

    View Slide

100. We have the assertion
     library for test just fine, but
     no mocking library yet
     

     View Slide

101. Concurrency on mruby
     Thread is not there, but fiber is there I think
     

     View Slide

102. Need to explore catching
     signals for graceful
     shutdown
     

     View Slide

103. Documentation of mruby is
     a bit lacking
     One area of improvement that we can all help
     

     View Slide

104. Thank You!
     Come greet me for questions (and stickers!!)
     Or tweet me @giosakti
     

     View Slide