From String undump to String unescape

Lightning talk on RubyKaigi 2018, Sendai, Japan.

Video: https://youtu.be/hCZWrvO_27k (in Japanese)
Moved/copied from:
* https://www.slideshare.net/tadd/from-stringundump-to-stringunescape
* https://niconare.nicovideo.jp/watch/kn3098

Transcript

1. From String#undump to String#unescape 斎藤ただしただし Tadashi Saito
2. Do you know String#undump ?
3. New (unknown) built-in method in Ruby 2.5 [bug#12275]
4. What?
5. "Does inverse of String#dump"
6. end
7. So…
8. Do you know String#dump ?
9. From the document: Produces a version of str with all non-printing characters replaced by \nnn notation and all special characters escaped.
10. In short: Escape to printable ASCII
11. Then,
12. String#undump (almost) does unescape from printable ASCII
13. Escape format example
14. 1. backslash + one character \n \t \\ 2. \x + 2 hex digits \x12 3. \u + 4 hex digits \u1234 etc etc...
15. because notations used in #dump is commonly used among various languages #undump is useful
16. I saw first these in C
17. Ruby Python Java C# JSON Nginx…
18. ＿人人人人人人人人人＿ ＞　Suddenly Nginx　＜ ￣Y^Y^Y^Y^Y^Y^Y^Y￣
19. In fact, Nginx was my trigger to implement String#undump
20. "We need to implement String#undump to read access log that Nginx writes"
21. Nginx module ngx_http_log_module (access_log, log_format) spec:
22. Access: /api/{URL-escaped-mbchars} Automatically!
23. Access: /api/{URL-escaped-mbchars} ↓ Internally: /api/{raw-mbchars} Automatically!
24. Access: /api/{URL-escaped-mbchars} ↓ Internally: /api/{raw-mbchars} ↓ Log: /api/{\xnn-escaped-mbchars} Automatically!
25. Access log looks like:
26. /api/\xe3\x81\x82\xe3\x81\x82\xe3\x81\x82\xe3\x81\x82 /api/\xe3\x81\x84\xe3\x81\x84\xe3\x81\x84\xe3\x81\x84 /api/\xe3\x81\x86\xe3\x81\x86\xe3\x81\x86\xe3\x81\x86 /api/\xe3\x81\x88\xe3\x81\x88\xe3\x81\x88\xe3\x81\x88 /api/\xe3\x81\x8a\xe3\x81\x8a\xe3\x81\x8a\xe3\x81\x8a /api/\xe3\x81\x8b\xe3\x81\x8b\xe3\x81\x8b\xe3\x81\x8b /api/\xe3\x81\x8d\xe3\x81\x8d\xe3\x81\x8d\xe3\x81\x8d /api/\xe3\x81\x8f\xe3\x81\x8f\xe3\x81\x8f\xe3\x81\x8f /api/\xe3\x81\x91\xe3\x81\x91\xe3\x81\x91\xe3\x81\x91 /api/\xe3\x81\x93\xe3\x81\x93\xe3\x81\x93\xe3\x81\x93 : : : :
27. I CAN NOT READ for analysis 😢
28. ＿人人人人人人人人人人人人人人＿ ＞ Can't change this behavior!! ＜ ￣Y^Y^Y^Y^Y^Y^Y^Y^Y^Y^Y^Y^Y￣ (at that time at least)
29. 💡 Decode with Ruby!
30. 1. Write it by myself
31. Easily done, but...
32. Too common problem to implement by myself?
33. 2. Google it for gems / codes
34. No gems 😢
35. Stack Overflow: Loose and partially incomplete codes 😢
36. 3. Final means: Google it in bugs.ruby-lang.org
37. Ticket found! [bug#12275]
38. •Submitted about a year ago •Specs (including the name – most difficult part) were almost fixed! 🙌
39. Matz: "If someone implement, it's OK to add."
40. But no one implements...
41. 💡 💡 Chance to commit!
42. I messed up and worked
43. Shipped as built-in String method in Ruby 2.5! 🎉
44. Thank you for involved committer / adviser (especially @naruse)
45. puts ‘/api/\xe3\x81\x82’.undump
46. #=> RuntimeError (invalid dumped string...)
47. • undump-ed string must be wrapped with "…" like ‘”\u1234”’ •Because String#dump always wraps with "..." String#undump Spec
48. Reasonable as spec of "inverse of #dump", but...
49. I have a little regret (to be honest)
50. •A little dull to prepend / append ” •Both of the original use-case and my case don't need this behavior
51. Plan to improve?
52. 1. Extend existing spec like str.undump(allow_no_quotes: true)
53. Too long!
54. 2. Introduce new method like str.just_unescape_backslashed_chars
55. Hard way to find good name...
56. So I created a (personal) gem
57. string_unescape https://github.com/tadd/string_unescape
58. if self[0] == '"' && self[-1] == '"’ self else ‘"’ + self + ‘”’ end.undump Definition
59. puts '/api/\xe3\x81\x82'.unescape #=> /api/あ
60. Easy ☺
61. Problems
62. •The name may have arguments – Hard to merge into the core •But just works™
63. •From business knowledge, •Found a potential and generic problem •Implemented as (core / external) method Summary of What I did
64. Please make use • String#undump (built-in) & • String#unescape (string_unescape gem) more!!1