Walking around the ruby forest more deeply

my presentation of RubyKaigi2014. for beginners, introducing about how to start to read Ruby implementation source code.

Transcript

1. YUKI TORII(SASADA)
   2014.10.19 RUBYKAIGI
   WALK AROUND THE RUBY FOREST

   MORE DEEPLY.
   THE GUIDE TO KNOW RUBY IMPLEMENTATION FOR NON-C LANGUAGE
   PROGRAMERS
   ϧϏΟϊεεϝ
   

   View Slide

2. WHO AM I?
   • ௗҪʢ࡫ాʣઇ / YUKI TORI
   • Rails application developer in Japan 

   (company: ສ༿ʣ
   • RailsGirls in JP organiser
   • Learned Programming by Rails
   

   View Slide

3. ADVERTISEMENT
   • Title:ϓϩϑΣογϣφϧͷͨΊͷ࣮ફHerokuೖ໳
   • Author:

   Ayumu Aizawa

   Arton

   Yuki Torii

   Keiko Oda
   

   View Slide

4. FIRST
   • This talk is for non C programmers/ beginners
   like me.
   • goal:

   After go back home tonight, you glance at
   Ruby’s source code.
   ͜ͷτʔΫ͸ॳ৺ऀ޲͚Ͱ͢

   ʢࢲΈ͍ͨͳʣ
   

   View Slide

5. FIRST
   • But for beginners, Ruby’s implementation is like
   a deep, dark forest…
   • You must get Maps and Compass.
   

   View Slide

6. THE PREPARATION TO WALK INTO
   THE RUBY INTERNAL
   FOREST
   ࢄาͷ४උ
   

   View Slide

7. PUT ON YOUR BOOTS:
   • Get a set of Ruby source

   

   • Put a first book in C-language beside of you

   (ask your friends or Google about good book.)

   > git clone https://github.com/ruby/ruby
   ϧϏʔͷιʔείʔυΛखʹೖΕΔ
   

   View Slide

8. MAP&COMPASS
   • RUBY UNDER A MICROSCOPE
   • RUBY HACKING GUIDE
   5XPCPPLT
   

   View Slide

9. MAP1:
   RUBY UNDER A MICROSCOPE
   • author: Pat Shaughnessy
   • ISBN: 9781593275273
   • an Illustrated Guide

   to Ruby Internals.
   

   View Slide

10. RUBY UNDER A MICROSCOPE 

    WILL GIVE YOU…
    • a large scale Map

    about 2.0 Ruby Implementation. 

    • Parsing, and Compilation of Ruby code
    • Structures of Ruby Basic objects
    • and etc…
    

    View Slide

11. MAP1: PARSE-COMPILE
    Class MyCode
    deff my
    …
    end
    !
    …
    end
    Your Ruby Code
    Perse Compile
    AST ByteCode
    Code-> ந৅ߏจ໦ -> ByteCode
    

    View Slide

12. MAP2:STRUCTURE OF OBJECTS
    String info… Array info…
    VALUE
    RSring
    flags
    klass
    RBasic
    VALUE
    RArray
    flags
    klass
    RBasic
    integer value Flags
    RObject built-in Object
    FIXNUM_FLAG : 1
    simple ruby value
    RObject
    numiv
    ivptr
    VALUE
    flags
    klass
    RBasic
    and RRegexp etc…

    built-in objects
    and Symbol ect…

    Simple values
    

    View Slide

13. JAPANESE VER. IS COMING
    SOON!
    • ʰRubyͷ͘͠Έ 

    Ruby Under a Microscopeʱ
    • translators:

    Koji Shimada 

    Shintaro Kakutani
    • ൃߦɿΦʔϜࣾ։ൃ෦
    • ץߦ༧ఆɿ2014೥ळ
    • I joins as one of reviewers.
    * ΋Ζ΋Ζௐ੔தͷͨΊ·ͩΦʔϜࣾʹ͸͓໰͍߹Θͤ͠ͳ͍Ͱ͍ͩ͘͞
    

    View Slide

14. MAP2: 

    RUBY HACKING GUIDE(RHG)
    • author: Minero Aoki
    • Original is written in Japanese,
    and is translated into English. 

    All contents are available at

    http://ruby-hacking-
    guide.github.io/
    • Talking about very details on
    Ruby source code and
    implementation.
    • about Ruby ver. 1.7.3…

    old, but Very Very Useful!!
    

    View Slide

15. WHY RHG IS USEFUL? 

    BECAUSE IT’S VERY UNIVERSAL.
    • “How to read source codes”
    • making a method simple by cutting not essential
    code off.
    • understanding object structure before.
    • and many other wisdoms
    • It was written for C-programmers.

    But non C-programmer can learn

    Basic C essence for reading Ruby Implementation.
    essence
    

    View Slide

16. WHY RHG IS USEFUL? 

    BECAUSE IT’S SOMEHOW OLD.
    • Ruby is improving.
    • While reading Ruby source code with RHG, you
    will find a difference between the description
    and the current code.

    => This is a chance to know Why the source has
    been changed. It’s a trace of improvement of
    ruby.

    You can trace it by ChangeLog and
    with its commit message.
    

    View Slide

17. MAP3: SOURCE FILE MAP
    BSDL COPYING COPYING.ja ChangeLog GPL KNOWNBUGS.rb LEGAL Makefile.in NEWS
    README.EXT README.EXT.ja README.ja.md README.md addr2line.c addr2line.h array.c
    benchmark bignum.c bin boots traptest class.c common.mk compar.c compile.c complex.c
    configure.in constant.h cont.c cygwin debug.c defs dir.c dln.c dln.h dln_find.c dmydln.c
    dmyext.c doc enc encoding.c enum.c en umerator.c error.c eval.c eval_error.c eval_intern.h
    eval_jump.c ext file.c gc.c gc.h gem_prelude.rb golf_prelude.rb goruby.c hash.c ia64.s include
    inits.c insns.def internal.h io.c iseq.c iseq.h lex.c.blt lib load.c loadpath.c localeinit.c main.c
    man marshal.c math.c method.h miniinit.c misc missing nacl node.c node.h numeric.c object.c
    pack.c parse.y prelude.rb probes.d probes_helper.h proc.c process.c random.c range.c
    rational.c re.c regcomp.c regenc.c regenc.h regerror.c regexec.c regint.h regparse.c regparse.h
    regsyntax.c ruby.c ruby_atomic.h safe.c sample signal.c siphash.c siphash.h sparc.c spec
    sprintf.c st.c strftime.c string.c struct.c symbian template test thread.c thread_native.h
    thread_pthread.c thread_pth read.h thread_win32.c thread_win32.h time.c timev.h tool
    transcode.c transcode_data.h util.c variable.c version.c version.h vm.c vm_backtrace.c
    vm_core.h vm_debug.h vm_dump.c vm_eval .c vm_exec.c vm_exec.h vm_insnhelper.c
    vm_insnhelper.h vm_method.c vm_opts.h vm_trace.c vsnprintf.c win32
    > cd ./ruby; ls
    ઈ๬….(desperate..)
    

    View Slide

18. MAP3: SOURCE FILE MAP
    BSDL COPYING COPYING.ja ChangeLog GPL KNOWNBUGS.rb LEGAL Makefile.in NEWS
    README.EXT README.EXT.ja README.ja.md README.md addr2line.c
    addr2line.h array.c benchmark bignum.c bin boots traptest class.c common.mk compar.c
    compile.c complex.c configure.in constant.h cont.c cygwin debug.c defs dir.c dln.c dln.h
    dln_find.c dmydln.c dmyext.c doc enc encoding.c enum.c en umerator.c error.c eval.c
    eval_error.c eval_intern.h eval_jump.c ext file.c gc.c gc.h gem_prelude.rb golf_prelude.rb
    goruby.c hash.c ia64.s include inits.c insns.def internal.h io.c iseq.c iseq.h lex.c.blt lib load.c
    loadpath.c localeinit.c main.c man marshal.c math.c method.h miniinit.c misc missing nacl
    node.c node.h numeric.c object.c pack.c parse.y prelude.rb probes.d probes_helper.h proc.c
    process.c random.c range.c rational.c re.c regcomp.c regenc.c regenc.h regerror.c regexec.c
    regint.h regparse.c regparse.h regsyntax.c ruby.c ruby_atomic.h safe.c sample signal.c siphash.c
    siphash.h sparc.c spec sprintf.c st.c strftime.c string.c struct.c symbian template test
    thread.c thread_native.h thread_pthread.c thread_pth read.h thread_win32.c thread_win32.h
    time.c timev.h tool transcode.c transcode_data.h util.c variable.c version.c version.h vm.c
    vm_backtrace.c vm_core.h vm_debug.h vm_dump.c vm_eval .c vm_exec.c vm_exec.h
    vm_insnhelper.c vm_insnhelper.h vm_method.c vm_opts.h vm_trace.c vsnprintf.c win32
    Don't give up your hope.
    File names are easy to understand.
    and README.EXT has

    “Appendix A. Ruby Source Files Overview”
    

    View Slide

19. MAP3: SOURCE FILE MAP
    ”Ruby Source Files Overview” in README.EXT
    groups source files by their contents.
    Ruby Language Core
    class.c :: classes and modules
    error.c :: exception classes and exception mechanism
    gc.c :: memory management
    load.c :: library loading
    object.c :: objects
    variable.c :: variables and constants
    Class Library
    array.c :: Array
    bignum.c :: Bignum
    compar.c :: Comparable
    complex.c :: Complex
    cont.c :: Fiber, Continuation
    dir.c :: Dir
    enum.c :: Enumerable
    enumerator.c :: Enumerator
    file.c :: File
    hash.c :: Hash
    io.c :: IO
    marshal.c :: Marshal
    math.c :: Math
    numeric.c :: Numeric, Integer, Fixnum, Float
    pack.c :: Array#pack, String#unpack
    proc.c :: Binding, Proc
    process.c :: Process
    random.c :: random number
    range.c :: Range
    rational.c :: Rational
    re.c :: Regexp, MatchData
    signal.c :: Signal
    sprintf.c :: String#sprintf
    string.c :: String
    struct.c :: Struct
    time.c :: Time
    Ruby Syntax Parser
    parse.y :: grammar definition
    parse.c :: automatically generated from parse.y
    keywords :: reserved keywords
    lex.c :: automatically generated from keywords
    Ruby Evaluator (a.k.a. YARV),
    Regular Expression Engine (Oniguruma),
    Utility Functions,
    Ruby Interpreter Implementation,
    Multilingualization,
    goruby Interpreter Implementation
    and other groups
    

    View Slide

20. MAP3: SOURCE FILE MAP
    File names are easy to understand.
    see the class libraries.
    Ruby Language Core
    class.c
    error.c
    gc.c
    load.c
    object.c
    variable.c
    Ruby Syntax Parser
    parse.y :: grammar definition
    parse.c :: automatically generated from parse.y
    keywords :: reserved keywords
    lex.c :: automatically generated from keywords
    Ruby Evaluator (a.k.a. YARV)
    Regular Expression Engine (Oniguruma)
    Utility Functions
    Ruby Interpreter Implementation
    Multilingualization
    goruby Interpreter Implementation
    and other groups
    Class Library
    array.c :: Array
    bignum.c :: Bignum
    compar.c :: Comparable
    complex.c :: Complex
    cont.c :: Fiber, Continuation
    dir.c :: Dir
    enum.c :: Enumerable
    enumerator.c :: Enumerator
    file.c :: File
    hash.c :: Hash
    io.c :: IO
    

    View Slide

21. MAP3: SOURCE FILE MAP
    extension also give us hints.
    xx.c => Of cause c code
    xx.h => header file. Many definitions such as macros,
    data-structures.
    xx.y => only parse.y (grammar definition)
    xx.rb => Of cause ruby file.
    xx.def => miscellaneous definitions
    

    View Slide

22. MAP3: SOURCE FILE MAP
    /include/ … many ruby’s core header files.
    /defs/ … definition files such as keywords, errors…
    /ext/ … c extensions such as json, socket…
    /lib/ …ruby libraries such as csv, irb, yaml, rake…
    /missing/ …implements of C functions need for ruby,

    but not all Systems have.
    /test/ … test codes for ruby implementations.
    /spec/ … RubySpec will be downloaded here.
    Special directories.
    

    View Slide

23. COMPASS:METHOD DEFINITION
    Because some file names correspond to class
    names,
    it is easy to find where a method is defined.
    What Hash#to_a does…?
    'PSFYBNQMF JGPOFEBZ*TUBSUUPXPOEFSJOH
    BOEDBOOPUTMFFQXFMM
    

    View Slide

24. COMPASS:METHOD DEFINITION
    Because some file names correspond to class
    name,
    it is easy to find where a method is defined.
    What Hash#to_a does…?
    'PSFYBNQMF JGPOFEBZ*TUBSUUPXPOEFSJOH
    BOEDBOOPUTMFFQXFMM
    See “hash.c”,
    Of course.
    and GoodNight.
    

    View Slide

25. COMPASS:METHOD DEFINITION
    Easy to find the definition of Hash#to_a in
    hash.c
    Because…
    !
    WPJE
    *OJU@)BTI WPJE
    
    \
    TOJQ
    !
    SC@EFpOF@NFUIPE SC@D)BTI JOJUJBMJ[F SC@IBTI@JOJUJBMJ[F
    
    SC@EFpOF@NFUIPE SC@D)BTI JOJUJBMJ[F@DPQZ SC@IBTI@JOJUJBMJ[F@DPQZ
    
    SC@EFpOF@NFUIPE SC@D)BTI SFIBTI SC@IBTI@SFIBTI
    
    !
    SC@EFpOF@NFUIPE SC@D)BTI UP@IBTI SC@IBTI@UP@IBTI
    
    SC@EFpOF@NFUIPE SC@D)BTI UP@I SC@IBTI@UP@I
    
    SC@EFpOF@NFUIPE SC@D)BTI UP@B SC@IBTI@UP@B
    
    SC@EFpOF@NFUIPE SC@D)BTI JOTQFDU SC@IBTI@JOTQFDU
    
    SC@EFpOF@BMJBT SC@D)BTI UP@T lJOTQFDU
    
    !
    TOJQNBOZPUIFSNFUIPEEFpOJUJPOT
    built-in class has Init_xx function

    at bottom of its file.
    }In the Init_xx,
    almost all methods
    are defined by
    “rb_define_method”
    

    View Slide

26. COMPASS:METHOD DEFINITION
    rb_define_method takes understandable args.
    TUBUJD7"-6&
    SC@IBTI@UP@B 7"-6&IBTI
    
    \
    7"-6&BSZ
    !
    BSZSC@BSZ@OFX@DBQB 3)"4)@4*;& IBTI
    
    
    SC@IBTI@GPSFBDI IBTI UP@B@J BSZ
    
    0#+@*/'&$5 BSZ IBTI
    
    !
    SFUVSOBSZ
    ^
    SC@EFpOF@NFUIPE SC@D)BTI UP@B SC@IBTI@UP@B
    
    klass method_name function init arg
    function name is put on the head of line

    to search easily.
    make new array with hash size
    convert hash’s each pair to array
    Macro often defined in ***.h.

    but I can imagine how work from name…
    

    View Slide

27. Good Night.
    

    View Slide

28. NOW, YOU GOT A READY.

    WOULD YOU LIKE TO

    WALK IN RUBY INTERNAL A LITTLE?
    

    View Slide

29. AN EXAMPLE ROUTE:

    ABOUT “HASH-TABLE”.
    

    View Slide

30. HASH TABLE IS USED
    EVERYWHERE.
    • Hash Class
    • Method Table
    • Instance Variables
    • Constant
    • Symbol
    HJUHSFQTU@JOJUcHSFQWEPDcHSFQWUFTU
    

    View Slide

31. WHAT IS A HASH TABLE.
    The data-structure for keeping key/value pair.
    obj of Hash class Method Table
    Key Value
    :favorite_animal “rabbit"
    “scores” [10,3,4]
    :neighborhood #
    Key Value
    “name” def name …
    “age” def age…
    “love?” def love?(me)….
    

    View Slide

32. OPEN HASH TABLE STRUCTURE
    bins entries
    ɾ
    !
    !
    ɾ
    “entry” under corresponding
    with hash-value in bins.
    

    View Slide

33. RUBY HASH TABLE STRUCTURE
    bins entries
    st_table
    type
    num_bins
    bins (big)
    num_entries
    st_table_entry
    hash
    key
    record
    next
    fore
    back
    1
    2
    3
    4
    next next
    fore back
    (hash is ordered

    from Ruby 1.9~)
    5
    6
    7
    entries_packed (0)
    back
    

    View Slide

34. RUBY HASH TABLE STRUCTURE
    entries
    st_packed_entry
    hash
    key
    value
    1
    2
    3
    4
    5
    st_table
    type
    num_bins
    packed
    num_entries
    entries_packed (1)
    real_entries
    

    View Slide

35. WHEN ENTRIES ARE TOO MANY.
    Packed -> bins
    1
    2
    3
    4
    5
    6
    key
    hash
    val
    

    View Slide

36. WHEN ENTRIES ARE TOO MANY.
    rehash(allocate more bins)
    (ਤͷentryͷ਺͸ద౰Ͱ͢…)
    

    View Slide

37. YOU CAN FIND ALL OF THEM IN
    • st_table

    in “include/ruby/st.h”
    • st_table_entry /st_packed_entry
    • add_packed_direct
    • unpacked
    • add_direct
    • rehash
    in “st.c”
    

    View Slide

38. LAST:DRAW YOUR OWN MAY

    BY YOUR HAND
    • The map of two books are a part of the rich
    Ruby forest.
    • You can find your favorite path, 

    or nice views by your self.
    ࣗ෼ͷ஍ਤΛ࡞ͬͯΈΑ͏
    

    View Slide

39. ENJOY YOUR RUBY LIFE.
    THANK YOU!
    

    View Slide