String meets Encoding

https://rubykaigi.org/2022/presentations/ima1zumi.html#day3

Transcript

1. String meets Encoding RubyKaigi 2022 2022-09-10 Mari Imaizumi
2. Agenda • Motivation • CSV.read • stackprof • String#split • perf • faster String#split • ruby/ruby #6351 2
3. Evaluation environments • MacBook Pro 2020 • macOS 12.4 • 2 GHz Quad-Core Intel Core i5 • 32 GB 3733 MHz LPDDR4X • Vagrant • Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-46-generic x86_64) • ruby 3.2.0dev (2022-09-05T15:39:37Z 63ed61e322) [x86_64-darwin21] 3
4. Introduction @ima1zumi (Mari Imaizumi) ESM, inc. Hamada.rb, Fukuoka.rb ❤ Character, Character Encoding 4
5. https://hamadarb.connpass.com/event/260134/ 5
7. Dive into Encoding - RubyKaigi Takeout 2021 7
8. Motivation • > If you want to do something around encoding in Ruby, you need to speed up String#encode. Right now it takes as long to convert CP932 to UTF-8 as it does to parse KEN_ALL.CSV in pure Ruby. (DeepL translate) • https://twitter.com/ktou/status/ 1436656477826019329 8
9. 🙆 String#encode 9
10. 🙆 String#encode 🤔 CSV.read (String#split) 10
11. CSV.read("KEN_ALL.CSV") 11
12. KEN_ALL.CSV • Zip code data in Japan • https://www.post.japanpost.jp/zipcode/dl/kogaki-zip.html • 16 MB • 15 lines • 124,541 rows • Encoding: CP932 (Windows-31J) 12
13. KEN_ALL.CSV 01101,060 ,0600000,ŴŕŜŘŪƄř,šŕŴƅƁŢŧœřśřŞ,ŘŜŬşŘšŘŜƄūŘŰƄŗŘ,๺ւಓ,ࡳຈࢢதԝ۠,ҎԼʹܝࡌ͕ ͳ͍৔߹,0,0,0,0,0,0 01101,064 ,0640941,ŴŕŜŘŪƄř,šŕŴƅƁŢŧœřśřŞ,ŗšűŜƄśŜ,๺ւಓ,ࡳຈࢢதԝ۠,Ѵέٰ,0,0,1,0,0,0 řŸ),๺ւಓ,ࡳຈࢢதԝ۠,๺Ұ৚੢ʢ̎̌ʙ̎̔ஸ໨ʣ,1,0,1,0,0,0 ...(about 120000 lines)... 47382,90718,9071800,śŝūƂşƃ,źŚźŵŞƄƃżūŞƄŬŧŔř,ŘŜŬşŘšŘŜƄūŘŰƄŗŘ,ԭೄݝ,ീॏࢁ܊༩ಹࠃொ,ҎԼʹ ܝࡌ͕ͳ͍৔߹,0,0,0,0,0,0 47382,90718,9071801,śŝūƂşƃ,źŚźŵŞƄƃżūŞƄŬŧŔř,żūŞƄŬ,ԭೄݝ,ീॏࢁ܊༩ಹࠃொ,༩ಹ ࠃ,0,0,0,0,0,0 13
14. Benchmark for CSV.read 14
15. Benchmark for CSV.read 15
16. stackprof 🔍 16
17. Stackprof • A sampling call-stack pro fi ler for Ruby • https://github.com/tmm1/stackprof • sampling mode • :wall, :cpu, :object, :custom • fl amegraph 17
18. Stackprof 18
20. stackprof --d3- fl amegraph stackprof-cpu-cp932- csv.dump > stackprof.html 20
21. Stackprof 21
22. grep split 22
23. grep split 23
24. Summary • Reading KEN_ALL.CSV with CSV.read took about 1.8 seconds. • CSV.read uses 29% for String#split 24
25. Measure String#split with perf 25
26. String#split • split(pattern = nil, limit = 0) • pattern: Regexp, String, nil • limit: number of splits • return: Array or self 26
27. Try perf 27 • performance analyzing tool in Linux
29. perf record String#split 29
31. fl amegraph 31 →alphabetical order
32. fl amegraph 32 rb_ary_push rb_enc_cr_str_copy_for_substr →alphabetical order str_new0
33. Summary • str_new0 40.13% • rb_ary_push 19.25% • rb_enc_cr_str_copy_for_substr 13% • rb_mem_search 4.88% • rb_enc_right_char_head 3.68% 33
34. String#split • 1. Check arguments • 2. Check patterns • 3. loop • 1. Search substr • 2. create substr • 3. result << substr • 4. return result 34
35. rb_str_split_m summary • str_new0 40.13% • rb_ary_push 19.25% • rb_enc_cr_str_copy_for_substr 13% • rb_mem_search 4.88% • rb_enc_right_char_head 3.68% 35
36. Summary • str_new0 40.13% • rb_ary_push 19.25% • rb_enc_cr_str_copy_for_substr 13% • rb_mem_search 4.88% • rb_enc_right_char_head 3.68% 36
37. rb_str_subseq 37
38. rb_str_subseq 38 create substring from str
39. rb_str_subseq 39 set encoding and coderange to str2 create substring from str
40. rb_ enc_ cr_ str_ copy_ for_ substr 40
41. rb_ enc_ cr_ str_ copy_ for_ substr 41 set encoding
42. rb_ enc_ cr_ str_ copy_ for_ substr 42 set encoding set coderange
43. str_enc_copy 43
46. 🤔 • Don't get encoding dynamically • just pass the Encoding of the original string 46
47. Make rb_enc_set_index_fastpath 47
48. Benchmark for String#split 48
49. Benchmark for String#split https://github.com/ruby/ruby/pull/6351 49 SVCZ
     SVCZ
    EFW
     CVJMUSVCZ 4USJOHTQMJU
    65'    4USJOHTQMJU
    64"4$**    SVCZ
    
    Y
    SVCZ
    EFW
    Y SVCZ
    
    Y
    SVCZ
    EFW
    Y
50. Conclusion • String, Encoding check is a bit heavy • must_encindex • mustnot_broken • Not checking or omitting unnecessary checks leads to faster speeds • https://github.com/ruby/ruby/pull/6072#issuecomment-1191371088 50