Slide 1

Slide 1 text

REQUIRE HOOKS Vladimir Dementyev Evil Martians Ruby拡張性の ⽋けたピースを 埋める

Slide 2

Slide 2 text

palkan_tula RUBY IS FLEXIBLE Rubyはプログラマを 制限しない⾔語なのです 2

Slide 3

Slide 3 text

Can I add a method to String? Stringにメソッドを追加できる? Just re-open a class クラスを再オープンすればいいよ

Slide 4

Slide 4 text

I don't want it a!ect everyone 全体に影響させたくない Refinements...

Slide 5

Slide 5 text

... or Ruby::Box REFINEMENTS??!!

Slide 6

Slide 6 text

Can I intercept a method on a class I don't own? ⾃分のものではないクラスのメソッドを 横取りできる? Module#prepend

Slide 7

Slide 7 text

Can I intercept a method that doesn't exist? まだ存在しないメソッドを横取りできる? #method_missing

Slide 8

Slide 8 text

Can I react when a class is defined? クラスが定義された時に反応できる? TracePoint.new(:end)

Slide 9

Slide 9 text

Can I intercept a Ruby file being loaded and transform it? Rubyファイルの読み込みを横取りして変 換できる? ...

Slide 10

Slide 10 text

Hm, #require is just a method but... # require もただのメソッドだけど…

Slide 11

Slide 11 text

palkan_tula HOW TO HOOK #require? # require を フックするには? 11

Slide 12

Slide 12 text

palkan_tula 12 github.com/palkan

Slide 13

Slide 13 text

palkan_tula 13 2017-... 2016-... 2015-... 2015-...

Slide 14

Slide 14 text

palkan_tula 14 2017-... 2016-... 2015-... 2015-...

Slide 15

Slide 15 text

palkan_tula WHY TO HOOK #require? なぜ#requireを フックする? 15

Slide 16

Slide 16 text

palkan_tula 16

Slide 17

Slide 17 text

palkan_tula 17 github.com/ruby-next/ruby-next

Slide 18

Slide 18 text

palkan_tula 18

Slide 19

Slide 19 text

CASE #1 NO BUILD TRANSPILING ソースコードを実⾏時にトランスパイルする 19 palkan_tula # some_boot.rb # Activate runtime transpling require "ruby-next/language/runtime" # All required files are transpiled on-the-fly require "environment" require "server" # ...

Slide 20

Slide 20 text

PROOF OF CONCEPT module Kernel module_function alias_method :require_without_ruby_next, :require def require(path) RubyNext.require(path) rescue => e warn "Failed to require '#{path}': #{e.message}" require_without_ruby_next(path) end end palkan_tula 20

Slide 21

Slide 21 text

palkan_tula 21 Freezolite

Slide 22

Slide 22 text

CASE #2 ZERO COMMENT FREEZING マジックコメント不要の⽂字列リテラルfreeze 22 palkan_tula

Slide 23

Slide 23 text

REPEAT YOURSELF 繰り返しは問題ない 23 palkan_tula module Kernel module_function alias_method :require_without_freezolite, :require def require(path) return require_without_freezolite(path) unless Freezolite.target?(path) RubyVM::InstructionSequence.compile_option = {frozen_string_literal: true} require_without_freezolite(path) ensure RubyVM::InstructionSequence.compile_option = {frozen_string_literal: false} end end

Slide 24

Slide 24 text

CASE #3 EVERY RAILS APPLICATION 繰り返しは本当に問題ない? 24 palkan_tula # zeitwerk/core_ext/kernel.rb module Kernel alias_method :zeitwerk_original_require, :require def require(path) if loader = Zeitwerk::Registry.autoloads.registered?(path) if path.end_with?(".rb") required = zeitwerk_original_require(path) loader.__on_file_autoloaded(path) if required required else loader.__on_dir_autoloaded(path) true end else # ... end end end

Slide 25

Slide 25 text

CASE #2026 EVERY RUBY APPLICATION オープニングキーノート⾒ましたか?最⾼でした 25 palkan_tula

Slide 26

Slide 26 text

palkan_tula IS RUBY'S #require MECHANISM FLEXIBLE ENOUGH? Rubyの#requireの仕組みは ⼗分に柔軟か? 26

Slide 27

Slide 27 text

palkan_tula CODE LOAD HOOKS IN THE WILD 他の⾔語のコード読み込みフック 27 SIDENOTE

Slide 28

Slide 28 text

PERL: @INC hooks palkan_tula 28 # Prepend "use strict" to every loaded file unshift @INC, sub { my ($self, $filename) = @_; # Find the file ourselves for my $dir (grep { !ref } @INC) { my $path = "$dir/$filename"; next unless -f $path; open my $fh, '<', $path or next; my $prepend = "use strict; use warnings;\n"; return (\$prepend, $fh); } return; # not found, let others hooks handle it };

Slide 29

Slide 29 text

PYTHON: sys.meta_path palkan_tula 29 import sys, importlib.abc class MyHook(importlib.abc.MetaPathFinder, importlib.abc.Loader): def find_spec(self, fullname, path, target=None): if should_handle(fullname): return importlib.util.spec_from_loader( fullname, self) return None # skip — let others handle it def exec_module(self, module): # load or transform source here ... sys.meta_path.insert(0, MyHook())

Slide 30

Slide 30 text

NODE.JS: registerHooks palkan_tula 30 import { readFileSync } from 'node:fs'; import { registerHooks } from 'node:module'; import coffeescript from 'coffeescript'; const extensionsRegex = /\.(coffee|litcoffee|coffee\.md)$/; function load(url, context, nextLoad) { if (extensionsRegex.test(url)) { const { source: rawSource } = nextLoad(url, { ...context, format: 'coffee' }); const transformedSource = coffeescript.compile(rawSource.toString(), url); return { format: getPackageType(url), shortCircuit: true, source: transformedSource, }; } return nextLoad(url, context); } registerHooks({ load });

Slide 31

Slide 31 text

COMMON PATTERNS Resolvers (path to path) Loaders (path to source or full hijack) Chaining palkan_tula 31 ソースパスの独⾃解決 パスからソースまたはバイトコードへの独⾃ロジック フックの連鎖

Slide 32

Slide 32 text

COMMON USE CASES Mocking Transforming Remote sources Encrypted sources palkan_tula 32 テストでのモックとスタブ ソースコードの変換 リモートソースからのコード読み込み 暗号化されたソースコードの読み込み

Slide 33

Slide 33 text

palkan_tula HOW TO BRING SIMILAR POWERS TO RUBY? 同じ⼒をRubyにどう持ち込む? 33

Slide 34

Slide 34 text

REQUIRE PHASES palkan_tula 34 Loaded? Resolve Read Parse Compile path/name absolute path no yes Skip Eval source AST bytecode #load

Slide 35

Slide 35 text

REQUIRE PHASES palkan_tula 35 Loaded? Resolve Read Parse Compile path/name absolute path no yes Skip Eval source AST bytecode #load Hook?

Slide 36

Slide 36 text

palkan_tula INTRODUCING require-hooks 36

Slide 37

Slide 37 text

gem "require-hooks" API for intercepting #require/#load/etc MRI, JRuby, TruffleRuby Passes* ruby/spec * No #load(path, wrap: ...) github.com/ruby-next/require-hooks 37 palkan_tula

Slide 38

Slide 38 text

38 palkan_tula RequireHooks.around_load(patterns: ["/gem/dir/*.rb"]) do |path, &block| puts "Loading #{path}" block.call.tap { puts "Loaded #{path}" } end RequireHooks.source_transform(patterns: ["/my_project/*.rb"], exclude_patterns: ["/my_project/vendor/*"]) do |path, source| source ||= File.read(path) "# frozen_string_literal: true\n#{source}" end RequireHooks.hijack_load(patterns: ["/my_project/*.rb"]) do |path, source| source ||= File.read(path) if defined?(RubyVM::InstructionSequence) RubyVM::InstructionSequence.compile(source) elsif defined?(JRUBY_VERSION) JRuby.compile(source) end end

Slide 39

Slide 39 text

39 palkan_tula RequireHooks.around_load(patterns: ["/gem/dir/*.rb"]) do |path, &block| puts "Loading #{path}" block.call.tap { puts "Loaded #{path}" } end RequireHooks.source_transform(patterns: ["/my_project/*.rb"], exclude_patterns: ["/my_project/vendor/*"]) do |path, source| source ||= File.read(path) "# frozen_string_literal: true\n#{source}" end RequireHooks.hijack_load(patterns: ["/my_project/*.rb"]) do |path, source| source ||= File.read(path) if defined?(RubyVM::InstructionSequence) RubyVM::InstructionSequence.compile(source) elsif defined?(JRUBY_VERSION) JRuby.compile(source) end end Around hooks allow wrapping of code loading (e.g., instrumentation) Aroundフックでコード読み込みを ラップ可能(例:計測)

Slide 40

Slide 40 text

40 palkan_tula RequireHooks.around_load(patterns: ["/gem/dir/*.rb"]) do |path, &block| puts "Loading #{path}" block.call.tap { puts "Loaded #{path}" } end RequireHooks.source_transform(patterns: ["/my_project/*.rb"], exclude_patterns: ["/my_project/vendor/*"]) do |path, source| source ||= File.read(path) "# frozen_string_literal: true\n#{source}" end RequireHooks.hijack_load(patterns: ["/my_project/*.rb"]) do |path, source| source ||= File.read(path) if defined?(RubyVM::InstructionSequence) RubyVM::InstructionSequence.compile(source) elsif defined?(JRUBY_VERSION) JRuby.compile(source) end end Source transformation hooks (ad hoc transpiling) ソース変換フック (アドホックなトランスパイル)

Slide 41

Slide 41 text

41 palkan_tula RequireHooks.around_load(patterns: ["/gem/dir/*.rb"]) do |path, &block| puts "Loading #{path}" block.call.tap { puts "Loaded #{path}" } end RequireHooks.source_transform(patterns: ["/my_project/*.rb"], exclude_patterns: ["/my_project/vendor/*"]) do |path, source| source ||= File.read(path) "# frozen_string_literal: true\n#{source}" end RequireHooks.hijack_load(patterns: ["/my_project/*.rb"]) do |path, source| source ||= File.read(path) if defined?(RubyVM::InstructionSequence) RubyVM::InstructionSequence.compile(source) elsif defined?(JRUBY_VERSION) JRuby.compile(source) end end Full control of what's being loaded 何が読み込まれるか完全に制御

Slide 42

Slide 42 text

REQUIRE HOOKS palkan_tula 42 Loaded? Resolve Read Compile path/name absolute path yes Skip Eval source AST bytecode #load no new source #source_transform Parse #hijack_load no yes #around_load

Slide 43

Slide 43 text

REQUIRE HOOKS palkan_tula 43 Loaded? Resolve Read Compile path/name absolute path yes Skip Eval source AST bytecode #load no new source #source_transform Parse #hijack_load no yes #around_load $LOAD_PATH.unshift(...) ruby-next

Slide 44

Slide 44 text

44 palkan_tula RequireHooks.around_load(patterns:) do |_path, &block| was_frozen=RubyVM::InstructionSequence.compile_option[:frozen_string_literal] RubyVM::InstructionSequence.compile_option={frozen_string_literal: true} block.call ensure RubyVM::InstructionSequence.compile_option={frozen_string_literal: was_frozen} end gem "freezolite" github.com/ruby-next/freezolite

Slide 45

Slide 45 text

45 palkan_tula RequireHooks.source_transform(patterns:) do |path, contents| RubyNext::Language.load_path(path, contents) end gem "ruby-next" github.com/ruby-next/ruby-next

Slide 46

Slide 46 text

OUT INTO THE WORLD palkan_tula 46

Slide 47

Slide 47 text

palkan_tula REQUIRE HOOKS REQUIRE HACKS requireフックにはハックが必要 47

Slide 48

Slide 48 text

3 HOOK MODES Kernel#require patch #load_iseq (MRI only) ??? palkan_tula 48

Slide 49

Slide 49 text

MRI's #load_iseq RubyVM::InstructionSequence#load_iseq Since 2.3.0 Path to bytecode Singleton callback palkan_tula 49

Slide 50

Slide 50 text

palkan_tula 50 Loaded? Resolve Read Compile path/name absolute path yes Skip Eval source AST bytecode #load no #load_iseq Parse nil iseq MRI's #load_iseq

Slide 51

Slide 51 text

gem "bootsnap" Speeds up a Ruby program boot Bytecode, JSON/YAML and $LOAD_PATH caching Monopolizes #load_iseq github.com/rails/bootsnap 51 palkan_tula

Slide 52

Slide 52 text

3 HOOK MODES Kernel#require patch #load_iseq (MRI only) Bootsnap patch (MRI only) palkan_tula 52 HACK #1

Slide 53

Slide 53 text

STRATEGY palkan_tula 53 MRI? yes Bootsnap? yes Bootsnap patch #load_iseq no Kernel patch no

Slide 54

Slide 54 text

palkan_tula 54 Loaded? Resolve Read Compile path/name absolute path yes Skip Eval source AST bytecode #load no #load_iseq Parse nil iseq How to #around_load? #around_load ???

Slide 55

Slide 55 text

55 palkan_tula module RequireHooks::LoadIseq def load_iseq(path) ctx = RequireHooks.context_for(path) ctx.run_around_load_callbacks(path) do iseq = if ctx.source_transform? || ctx.hijack? new_contents = ctx.perform_source_transform(path) hijacked = ctx.try_hijack_load(path, new_contents) if hijacked hijacked elsif new_contents RubyVM::InstructionSequence.compile(new_contents, path, path, 1) end end iseq ||= (defined?(super) ? super : RubyVM::InstructionSequence.compile_file(path)) iseq.eval RubyVM::InstructionSequence.compile("") end end end

Slide 56

Slide 56 text

56 palkan_tula module RequireHooks::LoadIseq def load_iseq(path) ctx = RequireHooks.context_for(path) ctx.run_around_load_callbacks(path) do iseq = if ctx.source_transform? || ctx.hijack? new_contents = ctx.perform_source_transform(path) hijacked = ctx.try_hijack_load(path, new_contents) if hijacked hijacked elsif new_contents RubyVM::InstructionSequence.compile(new_contents, path, path, 1) end end iseq ||= (defined?(super) ? super : RubyVM::InstructionSequence.compile_file(path)) iseq.eval RubyVM::InstructionSequence.compile("") end end end Always compile and eval

Slide 57

Slide 57 text

57 palkan_tula module RequireHooks::LoadIseq def load_iseq(path) ctx = RequireHooks.context_for(path) ctx.run_around_load_callbacks(path) do iseq = if ctx.source_transform? || ctx.hijack? new_contents = ctx.perform_source_transform(path) hijacked = ctx.try_hijack_load(path, new_contents) if hijacked hijacked elsif new_contents RubyVM::InstructionSequence.compile(new_contents, path, path, 1) end end iseq ||= (defined?(super) ? super : RubyVM::InstructionSequence.compile_file(path)) iseq.eval RubyVM::InstructionSequence.compile("") end end end Return empty iseq to let MRI finish the feature loading without loading the code 空のiseqを返して、コードを 読み込まずにMRIの feature読み込みを完了させる HACK #2

Slide 58

Slide 58 text

palkan_tula FULL OF HACKS. PERFORMANCE? ハックだらけ。パフォーマンスは⼤丈夫? 58

Slide 59

Slide 59 text

palkan_tula palkan RubyKaigi‘20 59 hyperfine 'ruby project/project.rb' 'HOOKS=idle ruby project/ project.rb' 'HOOKS=around ruby project/project.rb' Benchmark 1: ruby project/project.rb Time (mean ± σ): 2.755 s ± 0.033 s Benchmark 2: HOOKS=idle ruby project/project.rb Time (mean ± σ): 2.757 s ± 0.010 s Benchmark 3: HOOKS=around ruby project/project.rb Time (mean ± σ): 2.772 s ± 0.011 s Summary ruby project/project.rb ran 1.01 ± 0.01 times faster than HOOKS=idle ruby project/ project.rb 1.01 ± 0.01 times faster than HOOKS=around ruby project/ project.rb ➜ BENCHMARK: 12K FILES

Slide 60

Slide 60 text

palkan_tula palkan RubyKaigi‘20 60 hyperfine 'HOOKS=around ruby project/project.rb' \ 'REQUIRE_HOOKS_MODE=patch HOOKS=around ruby project/project.rb' Benchmark 1: HOOKS=around ruby project/project.rb Time (mean ± σ): 3.111 s ± 0.007 s Benchmark 2: REQUIRE_HOOKS_MODE=patch HOOKS=around ruby project/ project.rb Time (mean ± σ): 3.469 s ± 0.020 s Summary HOOKS=around ruby project/project.rb ran 1.13 ± 0.01 times faster than REQUIRE_HOOKS_MODE=patch HOOKS=around ruby project/project.rb ➜ #load_iseq vs Kernel#require

Slide 61

Slide 61 text

REQUIRE PHASES palkan_tula 61 Loaded? Resolve Read Parse Compile path/name absolute path no yes Skip Eval source AST bytecode

Slide 62

Slide 62 text

palkan_tula REQUIRE PHASES 62 Loaded? Resolve path/name absolute path no yes Skip $LOAD_PATH scan $LOADED_FEATURES check

Slide 63

Slide 63 text

palkan_tula REQUIRE PHASES 63 Loaded? Resolve path/name absolute path no yes Skip $LOAD_PATH scan $LOADED_FEATURES check

Slide 64

Slide 64 text

64 palkan_tula module RequireHooks::KernelPatch alias_method :require_without_require_hooks, :require def require(path) _, realpath = *$LOAD_PATH.resolve_feature_path(path) return require_without_require_hooks(path) unless realpath ctx = RequireHooks.context_for(realpath) return require_without_require_hooks(path) if ctx.empty? return false if $LOADED_FEATURES.include?(realpath) RequireHooks::KernelPatch.lock_feature(feature) do |loaded| return false if loaded $LOADED_FEATURES << realpath RequireHooks::KernelPatch.load(realpath) true end end end

Slide 65

Slide 65 text

65 palkan_tula module RequireHooks::KernelPatch alias_method :require_without_require_hooks, :require def require(path) _, realpath = *$LOAD_PATH.resolve_feature_path(path) return require_without_require_hooks(path) unless realpath ctx = RequireHooks.context_for(realpath) return require_without_require_hooks(path) if ctx.empty? return false if $LOADED_FEATURES.include?(realpath) RequireHooks::KernelPatch.lock_feature(feature) do |loaded| return false if loaded $LOADED_FEATURES << realpath RequireHooks::KernelPatch.load(realpath) true end end end Ruby implementations are not designed for that Rubyの実装はこのような使い⽅を 想定していない

Slide 66

Slide 66 text

PATCHING CAVEATS Array#include? can be slow (Ruby uses an internal index) Manually modifying $LOAD_FEATURES can cause expensive invalidation パッチ適⽤の注意点 66 palkan_tula Array#include?は遅くなりうる(内部インデックスで⾼速化しているため) $LOAD_FEATURESの⼿動変更はコストの⾼い無効化を引き起こす可能性がある

Slide 67

Slide 67 text

67 palkan_tula static st_table * get_loaded_features_index(const rb_box_t *box) { int i; VALUE features = box->loaded_features; const VALUE snapshot = box->loaded_features_snapshot; if (!rb_ary_shared_with_p(snapshot, features)) { /* The sharing was broken; something (other than us in rb_provide_feature()) modified loaded_features. Rebuild the index. */ st_foreach(box->loaded_features_index, loaded_features_index_clear_i, 0); VALUE realpaths = box->loaded_features_realpaths; VALUE realpath_map = box->loaded_features_realpath_map; VALUE previous_realpath_map = rb_hash_dup(realpath_map); rb_hash_clear(realpaths); rb_hash_clear(realpath_map); features = rb_ary_resurrect(features); for (i = 0; i < RARRAY_LEN(features); i++) { VALUE entry, as_str; as_str = entry = rb_ary_entry(features, i); StringValue(as_str); as_str = rb_fstring(as_str);

Slide 68

Slide 68 text

palkan_tula palkan RubyKaigi‘20 68 hyperfine 'ruby project/project.rb' \ 'HOOKS=around ruby project/project.rb' Benchmark 1: ruby project/project.rb Time (mean ± σ): 1.886 s ± 0.130 s Benchmark 2: HOOKS=around ruby project/project.rb Time (mean ± σ): 13.467 s ± 0.410 s Summary ruby project/project.rb ran 7.14 ± 0.16 times faster than HOOKS=around ruby project/ project.rb ➜ BENCHMARK: JRuby, 2.5k files

Slide 69

Slide 69 text

palkan_tula palkan RubyKaigi‘20 69 hyperfine 'ruby project/project.rb' \ 'HOOKS=around ruby project/project.rb' Benchmark 1: ruby project/project.rb Time (mean ± σ): 1.886 s ± 0.130 s Benchmark 2: HOOKS=around ruby project/project.rb Time (mean ± σ): 13.467 s ± 0.410 s Summary ruby project/project.rb ran 7.14 ± 0.16 times faster than HOOKS=around ruby project/ project.rb ➜ BENCHMARK: JRuby, 2.5k files JRuby rebuilds the index on #load when it detects outside changes to $LOADED_FEATURES JRubyは$LOADED_FEATURESの 外部変更を検知すると#load時に インデックスを再構築する

Slide 70

Slide 70 text

$LOADED_FEATURES+ Smarter $LOADED_FEATURES Faster lookup, less invalidations palkan_tula 70 より速い検索、より少ない無効化

Slide 71

Slide 71 text

$LOADED_FEATURES+ palkan_tula 71 HACK #3 Smarter $LOADED_FEATURES Faster lookup, less invalidations より速い検索、より少ない無効化

Slide 72

Slide 72 text

palkan_tula LAYERING HACKS IS NOT THE WAY ハックの積み重ねは正しい道ではない 72

Slide 73

Slide 73 text

palkan_tula GROUNDWORK, NOT PATCHWORK ⼟台を作れ、継ぎ接ぎするな 73

Slide 74

Slide 74 text

palkan_tula GROUNDWORK IDEAS 74 1. Promote $LOAD_FEATURES from Array to a full-featured object (w/ check and add APIs) 2. Introduce Ruby::Loader.register_hook and Ruby::Loader::Hook interface $LOAD_FEATURESをArrayから本格的なオブジェクトに昇格(checkとaddのAPIを持つ) Ruby::Loader.register_hookとRuby::Loader::Hookインターフェースの導⼊

Slide 75

Slide 75 text

GROUNDWORK IDEAS 0. Adopt require-hooks, stop patching Kernel yourself 1. Promote $LOAD_FEATURES from Array to a full-featured object (w/ check and add APIs) 2. Introduce Ruby::Loader.register_hook and Ruby::Loader::Hook interface palkan_tula 75 require-hooksを採⽤して、⾃分でKernelをパッチするのをやめよう $LOAD_FEATURESをArrayから本格的なオブジェクトに昇格(checkとaddのAPIを持つ) Ruby::Loader.register_hookとRuby::Loader::Hookインターフェースの導⼊

Slide 76

Slide 76 text

palkan_tula RUBY IS FLEXIBLE* *HACKS MAY BE REQUIRED Rubyは柔軟(ハックが必要な場合あり) 76

Slide 77

Slide 77 text

palkan_tula RUBY CAN BE FLEXIBLE* Rubyは柔軟になれる 77

Slide 78

Slide 78 text

palkan_tula INTRODUCING require-profiler 78 BONUS

Slide 79

Slide 79 text

gem "require-profiler" Profile your application boot process Visualize #require trees with Speedscope Know the most requiring gems/folders and who requires who github.com/palkan/require-profiler 79 palkan_tula アプリの起動をプロファイルできる Speedscopeで#requireツリーを可視化できる どのgem・フォルダが重く、何が何をrequireしているかを把握できる

Slide 80

Slide 80 text

palkan_tula 80 github.com/palkan/require-profiler

Slide 81

Slide 81 text

THANK YOU x.com/palkan_tula x.com/evilmartians ありがとう