Namespace, What and Why

The presentation deck for RubyKaigi 2024 about "Namespace on read".

Transcript

1. @tagomoris (Satoshi Tagomori) 
 RubyKaigi 2024 @ May 15, 2024 Namespace, What and Why Introducing “Namespace on Read”
2. @tagomoris Satoshi Tagomori Maintainer/Founder: OSS: Fluentd, MessagePack, 
 Norikra, Woothee, …
 Event: ISUCON Service: Pathtraq
3. https://www. fl ickr.com/photos/takkanm/3978417669 Asakusa.rb
4. Namespace on read https://bugs.ruby-lang.org/issues/19744
5. Who does “Namespace on read” consist on? Special Thanks to Matz, and …
6. Special Thanks to Matz, and … Great advisors and motivators Chris Saltzberg
 @shioyama Yusuke Endoh
 @mametter Yuichiro Kaneko
 @spikeolaf Hiroshi SHIBATA
 @hsbt Koichi Sasada
 @ko1 Misaki Shioi
 @coe401_
7. What is Namespace?
8. What is Namespace? Separates apps/libs
 into isolated “spaces”
 (in this session)
9. Namespace Separating apps/libs into isolated spaces • Load apps/libs in a space • Hide changes from apps/libs in a namespace to other spaces • Run methods de fi ned in a space with de fi nitions in the space
10. Before Namespace: Global Only All classes/modules are shared in the entire Ruby process Ruby Process Application Code App::Func User Library Code DB::Client (v2) Library Code ActiveSupport (v7)
11. Collision: 2 versions of 1 library cause errors Ruby Process Library Code DB::Client (v3) 😵 Application Code App::Func User Library Code DB::Client (v2) Library Code ActiveSupport (v7) Before Namespace: Global Only
12. Ruby Process Namespace Load apps/libs in a space Namespace Application Code App::Func User Library Code DB::Client (v2) Library Code ActiveSupport (v7)
13. Ruby Process Namespace Hide changes from apps/libs in a namespace to other spaces Namespace Application Code App::Func User Library Code DB::Client (v2) Library Code ActiveSupport (v7) Library Code DB::Client (v3) 😀
14. DEMO
15. 👏👏👏👏👏
16. Ruby Process Namespace Run methods de fi ned in a space with de fi nitions in the space Namespace Application Code App::Func User Library Code DB::Client (v2) Library Code ActiveSupport (v7) Namespace Library Code DB::Client (v3) Namespace Application Code App::Func2 User Library Code ActiveSupport (v6) Application Code call call call
17. Namespace (revisit) Separating apps/libs into isolated spaces • Load apps/libs in a space • Hide changes from apps/libs in a namespace to other spaces • Run methods de fi ned in a space with de fi nitions in the space
18. Why do we need Namespace?
19. Why do we need Namespace? To avoid collisions of
 names, de fi nitions, and versions
20. Name Collisions No one can use a name in two ways • Structured names • Foo::Bar::Baz • Major scenario: • Top-level “User” or “Configuration” classes (gems, apps, etc) • Two apps in a process (in the context of “Modular monolith”)
21. Definition Collisions De fi nitions can be modi fi ed, globally, from anywhere • A single Module/Class instance • Example: Oj.default_options = {symbol_keys: true} • A single Module/Class definition • Process global open classes (monkey patches) • A single set of constants • A single set of global variables
22. Version Collisions Two di ff erent versions of a library can’t be loaded • Application libraries and library dependencies • App uses X, and Y ver 2 • X depends on Y ver 1 • “Dependency hell” • App uses A and B • A depends on C ver 1 • B depends on C ver 2 App C ver 1 C ver 2 A B App Y ver 1 X Y ver 2
23. Why is the Namespace “on read”?
24. Why is the Namespace “on read”? To keep using tons of
 fantastic rubygems as-is!
25. Namespace “on read” “on read”: apps/libs into spaces when loaded/required • “on write” approach • Libraries (gems) need to be written in the namespace manner (on write) • Update all gems in rubygems…? 😱 • “on read” approach • Libraries/Apps are loaded into spaces when loaded/require (on read) • We can use all gems as-is, with bene fi ts of namespaces
26. How can we implement Namespace?
27. How can we implement Namespace? De fi ning classes/modules under a namespace
 Loading extensions locally
 Having class/module de fi nitions per namespace
28. Defining Classes/Modules Under a Namespace (NS) • Namespace as subclass of Module • Use existing mechanism of load(file, module) • Load .rb fi les (A::B) under a module (M) → M::A::B • See also: “Multiverse Ruby” (RubyKaigi 2023, @shioyama) • Modify C API to de fi ne class/module: class_alloc (in class.c) • Create classes/modules (X) under a namespace (NS) → NS::X • To support extensions (.so/.dll/.bundle)
29. Loading Extensions Locally Extensions invisible globally, accessible explicitly • Extension fi les can be loaded 2 times or more • Loading extensions (.so/.dll/.bundle) in namespace • Copying the extension fi le to a tmp dir with PID & namespace pre fi x:
 my_ext.so → 515_160_my_ext.so • Using RTLD_LOCAL only when RUBY_NAMESPACE=1 • Hide exported (C) symbols to other extensions
30. Having Class/Module Definitions Per namespace • Open class in NS can modify built-in classes/modules • For monkey patches in namespace: class Object; def blank?; … • In other NSs: Object without #blank? • Class/Module de fi nition: classext (struct: rb_classext_t) • super, method table, constant table, instance variables, subclasses, … • Switching classext realizes a class with di ff erent de fi nitions! 🤪
 (Thanks to @_ko1 and @mametter)
31. Copy on Write of classext Having Class/Module Definitions Built-in Class/Module User’s Class/Module Class/Module Example Object String Kernel JSON ActiveSupport MyAppUser Root Namespace ✅ ✅ ✅ Main
 Namespace Namespace
 A Namespace
 B Before Running User Scripts ✅ classext readable/writable
32. Copy on Write of classext Having Class/Module Definitions Built-in Class/Module User’s Class/Module Class/Module Example Object String Kernel JSON ActiveSupport MyAppUser Root Namespace 🔒 🔒 🔒 Main
 Namespace ✅ ✅ ✅ ✅ Namespace
 A Namespace
 B Running User Scripts (without NS) ✅ classext readable/writable CoW 🔒 classext read only Read unchanged one
33. Copy on Write of classext Having Class/Module Definitions Running User Scripts with NSs ✅ classext readable/writable Built-in Class/Module User’s Class/Module Class/Module Example Object String Kernel JSON ActiveSupport MyAppUser Root Namespace 🔒 🔒 🔒 Main
 Namespace ✅ ✅ ✅ ✅ Namespace
 A ✅ ✅ ✅ Namespace
 B 🔒 classext read only CoW CoW Read
34. Copy on Write of classext Having Class/Module Definitions Running User Scripts with NSs ✅ classext readable/writable Built-in Class/Module User’s Class/Module Class/Module Example Object String Kernel JSON ActiveSupport MyAppUser Root Namespace 🔒 🔒 🔒 Main
 Namespace ✅ ✅ ✅ ✅ Namespace
 A ✅ ✅ ✅ Namespace
 B ✅ ✅ 🔒 classext read only CoW Read
35. Where to go with Namespace in the future?
36. Where to go with Namespace in the future? JUST IDEAS, 
 NOT IMPLEMENTED YET
37. Where to go with Namespace in the future? Modular-monolith(-ish) deployment
 Goodbye Dependency hell
 Safe library loading: “Packages”
38. Modular-Monolith(-ish) Deployment Apps deployed on a (possible) app server • Applications server based on Namespace • namespace per app • multiple apps are mounted using namespace • 2 or more applications on a single tenant (process) • Without con fl icts/collisions by namespace • Especially for development without containers on laptops App Server Namespace Web app Request
 Response Routing
39. Namespace Goodbye Dependency Hell With (possible) Bundler integration • One is in namespace If there is a dependency con fl ict • Your apps can be deployed before the gem’s update! • IMPORTANT: • We MUST keep the “healthy, updated” rubygems society • How to avoid moral hazard? App C ver 1 C ver 2 A B NEED DISCUSSIONS
40. “Packages” API (tentative) All libraries are required in namespace! (really?) • Add a method to require gem as package, with implicit namespace p1 = require_package(‘my-awesome-client’) MyAwesomeClient = p1.export(:MyAwesomeClient) • Optional: default exports (in gemspec?) require_package ‘my-awesome-client’ MyAwesomeClient #=> NS::MyAwesomeClient • Optional: export-as require_package ‘my-awesome-client’, as: :MAClient MAClient #=> NS::MyAwesomeClient NEED DISCUSSIONS
41. When can we use Namespace?
42. When can we use Namespace? 🤔
43. When can we use Namespace? Still in development - Stay tuned! Thank you for listening!