Secrets of Regexp

Regexp class is in every Rubyist's toolbox. But do you know the theory behind it, and what goes on under the hood?

Transcript

1. Secrets of Regexp Hiro Asari Red Hat, Inc.
2. Let's Talk About Regular Expressions
3. Let's Talk About Regular Expressions • There is no regular expression
4. Let's Talk About Regular Expressions • A good approximation as a name
5. Let's Talk About Regexp
6. Some people, when confronted with a problem, think, "I know, I'll use regular expressions." Now they have two problems. Jaime Zawinski 12 Aug, 1997 http://regex.info/blog/2006-09-15/247 http://www.codinghorror.com/blog/2008/06/regular-expressions-now-you-have-two- problems.html The point is not so much the evils of regular expressions, but the evils of overuse of it.
7. Formal Language Theory • The Language L • Over Alphabet Σ
8. Formal Language Theory • Alphabet Σ={a, b, c, d, e, …, z, λ} (example)
9. Formal Language Theory • Alphabet Σ={a, b, c, d, e, …, z, λ} (example) • Words over Σ: "a", "b", "ab", "aequafdhfad"
10. Formal Language Theory • Alphabet Σ={a, b, c, d, e, …, z, λ} (example) • Words over Σ: "a", "b", "ab", "aequafdhfad" • Σ*: The set of all words over Σ
11. Formal Language over Σ • A subset L of Σ* (with various properties) • L can be finite, and enumerate well-formed words, but often infinite
12. Example • Language L over Σ = {a,b} • 'a' is a word • a word may be obtained by appending 'ab' to an existing word • only words thus formed are legal
13. a aab aabab Well-formed words
14. b aaaab abb Ill-formed words
15. Succinctly… • a(ab)*
16. Expression • Textual representation of the formal language against which an input is tested whether it is a well-formed word in that language
17. Regular Languages • ∅ (empty language) is regular
18. Regular Languages • ∅ (empty language) is regular • For each a ∈ Σ (a belongs to Σ), the singleton language {a} is a regular language.
19. Regular Languages • ∅ (empty language) is regular • For each a ∈ Σ (a belongs to Σ), the singleton language {a} is a regular language. • If A and B are regular languages, then A ∪ B (union), A•B (concatenation), and A* (Kleene star) are regular languages
20. Regular Languages • ∅ (empty language) is regular • For each a ∈ Σ (a belongs to Σ), the singleton language {a} is a regular language. • If A and B are regular languages, then A ∪ B (union), A•B (concatenation), and A* (Kleene star) are regular languages • No other languages over Σ are regular.
21. Regular Expressions • Expressions of regular languages
22. Regular Expressions • Expressions of regular languages Not
23. Regular? Expressions • It turns out that some expressions are more powerful and expresses non-regular languages • Language of 'squares': (.*)\1 • a, aa, aaaa, WikiWiki
24. How does Regexp work? • Build a finite state automaton representing a given regular expression • Feed the String to the regular expression and see if the match succeeds
25. a a
26. ab* a b
27. .* .
28. a$ a $
29. a? a ε
30. a|b a b
31. (ab|c) c a b
32. (ab+|c) c a b b
33. Match is attempted at every character, left to right
34. zyxwvutsrqponmlkjihgfedcba ^ /a$/ Regexp does not think, 'a$' can match only at the end of the line, so we should fast forward to the end of the line
35. zyxwvutsrqponmlkjihgfedcba ^ zyxwvutsrqponmlkjihgfedcba ^ /a$/ Regexp does not think, 'a$' can match only at the end of the line, so we should fast forward to the end of the line
36. zyxwvutsrqponmlkjihgfedcba ^ zyxwvutsrqponmlkjihgfedcba ^ zyxwvutsrqponmlkjihgfedcba ^ /a$/ Regexp does not think, 'a$' can match only at the end of the line, so we should fast forward to the end of the line
37. zyxwvutsrqponmlkjihgfedcba ^ zyxwvutsrqponmlkjihgfedcba ^ zyxwvutsrqponmlkjihgfedcba ^ zyxwvutsrqponmlkjihgfedcba ^ /a$/ Regexp does not think, 'a$' can match only at the end of the line, so we should fast forward to the end of the line
38. zyxwvutsrqponmlkjihgfedcba ^ zyxwvutsrqponmlkjihgfedcba ^ zyxwvutsrqponmlkjihgfedcba ^ zyxwvutsrqponmlkjihgfedcba ^ ⋮ zyxwvutsrqponmlkjihgfedcba ^ /a$/ Regexp does not think, 'a$' can match only at the end of the line, so we should fast forward to the end of the line
39. abc d a dfadg ^ abc d a dfadg ^ abc d a dfadg ^ abc d a dfadg ^ # matches 'abc d a dfadg ' ^\s*(.*)\s*$
40. def pathological(n=5) Regexp.new('a?' * n + 'a' * n) end 1.upto(40) do |n| print n, ": " print Time.now, "\n" if 'a'*n =~ pathological(n) end a?a?a?…a?aaa…a
41. aaa ^ a?a?a?aaa
42. Regexp tips
43. UP_TO_256 = /\b(?:25[0-5] # 250-255 |2[0-4][0-9] # 200-249 |1[0-9][0-9] # 100-199 |[1-9][0-9] # 2-digit numbers |[0-9]) # single-digit numbers \b/x IPV4_ADDRESS = /#{UP_TO_256}(?:\.#{UP_TO_256}){3}/ Use /x
44. \A, \z for strings ^, $ for lines • \A: the beginning of the string • \z: the end of the string • ^: after \n • $: before \n
45. always in Ruby \A, \z for strings ^, $ for lines • \A: the beginning of the string • \z: the end of the string • ^: after \n • $: before \n
46. What's the problem? also note the difference in what /m means
47. #! /usr/bin/env perl $a = "abc\ndef"; if ($a =~ /^d/) { print "yes\n"; } if ($a =~ /^d/m) { print "yes now\n"; } # prints 'yes now' What's the problem? also note the difference in what /m means
48. #! /usr/bin/env ruby a = "abc\ndef"; if (a =~ /^d/) p "yes" end What's the problem? http://guides.rubyonrails.org/security.html#regular-expressions
49. class File < ActiveRecord::Base validates :name, :format => /^[\w\.\-\+]+$/ end Security Implications http://guides.rubyonrails.org/security.html#regular-expressions
50. file.txt%0A<script>alert(‘hello’)</script>
51. file.txt%0A<script>alert(‘hello’)</script>
52. file.txt\n<script>alert(‘hello’)</script>
53. file.txt\n<script>alert(‘hello’)</script> /^[\w\.\-\+]+$/
54. file.txt\n<script>alert(‘hello’)</script> /^[\w\.\-\+]+$/ Match succeeds ActiveRecord validation succeeds
55. file.txt\n<script>alert(‘hello’)</script> /\A[\w\.\-\+]+\z/
56. file.txt\n<script>alert(‘hello’)</script> /\A[\w\.\-\+]+\z/ Match fails ActiveRecord validation fails
57. require 'benchmark' # simple benchmark for alternations and character class n = 5_000 str = 'cafebabedeadbeef'*5_000 Benchmark.bmbm do |x| x.report('alternation') do str =~ /^(a|b|c|d|e|f)+$/ end x.report('character class') do str =~ /^[a-f]+$/ end end Prefer Character Class to Alterations
58. Ruby 1.8.7 user system total real alternation 0.030000 0.010000 0.040000 ( 0.036702) character class 0.000000 0.000000 0.000000 ( 0.004704) Ruby 2.0.0 user system total real alternation 0.020000 0.010000 0.030000 ( 0.023139) character class 0.000000 0.000000 0.000000 ( 0.009641) JRuby 1.7.4.dev user system total real alternation 0.030000 0.000000 0.030000 ( 0.021000) character class 0.010000 0.000000 0.010000 ( 0.007000) Benchmarks
59. # case-insensitively match any non-word character… # one is unlike the others 'r' =~ /(?i:[\W])/ 's' =~ /(?i:[\W])/ 't' =~ /(?i:[\W])/ Beware of Character Classes matches, even if 's' is a word character https://bugs.ruby-lang.org/issues/4044
60. /^1?$|^(11+?)\1+$/
61. /^1?$|^(11+?)\1+$/ Matches '1' or ''
62. /^1?$|^(11+?)\1+$/ Non-greedily match 2 or more 1's
63. /^1?$|^(11+?)\1+$/ 1 or more additional times
64. /^1?$|^(11+?)\1+$/ matches a composite number
65. /^1?$|^(11+?)\1+$/ Matches a string of 1's if and only if there are a non-prime # of 1's
66. class Integer def prime? "1" * self !~ /^1?$|^(11+?)\1+$/ end end Integer#prime? No performance guarantee Attributed a Perl hacker Abigail
67. • @hiro_asari • Github: BanzaiMan