Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
More than Regexp
Search
Zete
November 18, 2012
Programming
16
1.2k
More than Regexp
RubyConfChina 2012 超越正则表达式的正则表达式
Zete
November 18, 2012
Tweet
Share
More Decks by Zete
See All by Zete
Thread Mist
luikore
7
740
Intro to Rust Programming Language
luikore
2
230
GC in Ruby 2.2
luikore
0
160
Building an Async Server with Fiber
luikore
3
740
Ruby 2.1 Walk Thru (title bait)
luikore
14
7.7k
Other Decks in Programming
See All in Programming
CloudNativePGを布教したい
nnaka2992
0
120
複数のAWSアカウントから横断で 利用する Lambda Authorizer の作り方
tc3jp
0
130
From the Wild into the Clouds - Laravel Meetup Talk
neverything
0
180
iOSでQRコード生成奮闘記
ktcryomm
2
130
Rubyと自由とAIと
yotii23
6
1.9k
color-scheme: light dark; を完全に理解する
uhyo
7
510
クックパッド検索システム統合/Cookpad Search System Consolidation
giga811
0
140
AIプログラミング雑キャッチアップ
yuheinakasaka
20
5.2k
なぜイベント駆動が必要なのか - CQRS/ESで解く複雑系システムの課題 -
j5ik2o
14
4.8k
自力でTTSモデルを作った話
zgock999
0
120
データベースのオペレーターであるCloudNativePGがStatefulSetを使わない理由に迫る
nnaka2992
0
250
メンテが命: PHPフレームワークのコンテナ化とアップグレード戦略
shunta27
0
320
Featured
See All Featured
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
Designing for humans not robots
tammielis
250
25k
Why You Should Never Use an ORM
jnunemaker
PRO
55
9.2k
StorybookのUI Testing Handbookを読んだ
zakiyama
28
5.5k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
356
29k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
33
2.8k
What's in a price? How to price your products and services
michaelherold
244
12k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.7k
Statistics for Hackers
jakevdp
797
220k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
233
17k
The Cult of Friendly URLs
andyhume
78
6.2k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.7k
Transcript
More Than Regexp https://github.com/luikore Monday, November 19, 12
Who m i Monday, November 19, 12
I mastered Basic, C, C++, Objective-C, Java, C#, Mathematica, Ruby,
Perl, Python, CoffeeScript, Haskell, Scala, Groovy, R, SML, Erlang, F#, MASM / GNU assembly, LLVM assembly Monday, November 19, 12
I mastered Basic, C, C++, Objective-C, Java, C#, Mathematica, Ruby,
Perl, Python, CoffeeScript, Haskell, Scala, Groovy, R, SML, Erlang, F#, MASM / GNU assembly, LLVM assembly ’s hello world Monday, November 19, 12
I know quite a lot about web development technologies, compiler
techniques, functional programing, quantum mechanics, abstract algebra, category theory and medieval history Monday, November 19, 12
I know quite a lot about web development technologies, compiler
techniques, functional programing, quantum mechanics, abstract algebra, category theory and medieval history a little bit Monday, November 19, 12
I am a rubyist just like you Monday, November 19,
12
on Languages Monday, November 19, 12
Parsing Parsing is reading, the computer way Monday, November 19,
12
Regexp The built-in parsing tool of Ruby Monday, November 19,
12
★ Search and replace Monday, November 19, 12
★ Search and replace ★ Validate form data Monday, November
19, 12
★ Search and replace ★ Validate form data ★ Implement
protocols Monday, November 19, 12
★ Search and replace ★ Validate form data ★ Implement
protocols ★ Virus scan Monday, November 19, 12
★ Search and replace ★ Validate form data ★ Implement
protocols ★ Virus scan ★ Matching mRNA motif Monday, November 19, 12
★ Search and replace ★ Validate form data ★ Implement
protocols ★ Virus scan ★ Matching mRNA motif ★ ... Monday, November 19, 12
A Brief History Monday, November 19, 12
★ Stephen Cole Kleene, 1950s ★ Ken Thompson, ed, g/re/p
★ Larry Wall, Perl Monday, November 19, 12
★ Stephen Cole Kleene, 1950s ★ Ken Thompson, ed, g/re/p
★ Larry Wall, Perl Monday, November 19, 12
★ Stephen Cole Kleene, 1950s ★ Ken Thompson, ed, g/re/p
★ Larry Wall, Perl Monday, November 19, 12
★ Stephen Cole Kleene, 1950s ★ Ken Thompson, ed, g/re/p
★ Larry Wall, Perl Monday, November 19, 12
★ Stephen Cole Kleene, 1950s ★ Ken Thompson, ed, g/re/p
★ Larry Wall, Perl Monday, November 19, 12
Modern Implementations Goes far beyond the original definition Monday, November
19, 12
★ Looks around Monday, November 19, 12
★ Looks around ★ Unicode support Monday, November 19, 12
★ Looks around ★ Unicode support ★ Matching history Monday,
November 19, 12
★ Looks around ★ Unicode support ★ Matching history ★
PEG engine in fact Monday, November 19, 12
★ Looks around ★ Unicode support ★ Matching history ★
PEG engine in fact ★ Ruby 1.9 Oniguruma (َ⻋车) Monday, November 19, 12
★ Looks around ★ Unicode support ★ Matching history ★
PEG engine in fact ★ Ruby 1.9 Oniguruma (َ⻋车) ★ Ruby 2.0 Onigmo (َӠ) Monday, November 19, 12
Regexp in Ruby RUBY_VERSION =~ /1\.9|2\.0/ s = “肿㜮办?” s[/肿㜮/]
= “ዎ㜮” #=> “ዎ㜮办?” %r(#{words.join ‘|’}) Monday, November 19, 12
foo “foo” exactly Monday, November 19, 12
foo “foo” exactly . matches ANY char Monday, November 19,
12
foo “foo” exactly . matches ANY char a|b or Monday,
November 19, 12
foo “foo” exactly . matches ANY char a|b or a?
maybe yes, maybe no Monday, November 19, 12
foo “foo” exactly . matches ANY char a|b or a?
maybe yes, maybe no a* kleeeeeene star Monday, November 19, 12
foo “foo” exactly . matches ANY char a|b or a?
maybe yes, maybe no a* kleeeeeene star a{0} repeat by 0 times Monday, November 19, 12
(a) group Monday, November 19, 12
(a) group \1 back ref (fixed) Monday, November 19, 12
(a) group \1 back ref (fixed) (?<name>a) define named group
Monday, November 19, 12
(a) group \1 back ref (fixed) (?<name>a) define named group
\g<name> use named ref Monday, November 19, 12
Difference between back ref and named group backref = /(\w+)
\1/ backref =~ ‘ha ha’ # 0 backref =~ ‘ha ho’ # false named = /(?:<word>\w+) \g<word>/ named =~ ‘ha ha’ # 0 named =~ ‘ha ho’ # 0 Monday, November 19, 12
★ Complex regexp contains much information Monday, November 19, 12
★ Complex regexp contains much information ★ Add space to
make it human-readable Monday, November 19, 12
★ Complex regexp contains much information ★ Add space to
make it human-readable ★ Try not to make too-complex regexps Monday, November 19, 12
What does it do? /^[ \t]*(?:class)\s*(.*?) \s*(<.*?)?\s*(#.*)?$/ Monday, November 19,
12
Add margins and paddings /^ [\ \t]* (?:class)\s* (.*?)\s* (<.*?)?\s*
(#.*)? $/x Monday, November 19, 12
Alignment reduces visual complexity: /^ [\ \t]* (?:class) \s* (.*?)
\s* (<.*?)? \s* (#.*)? $/x Monday, November 19, 12
Add comments r = /^ [\ \t]* (?:class) \s* (.*?)
\s* # class name (<.*?)? \s* # inheritance (#.*)? # line comment $/x r =~ “class A < B # match!” Monday, November 19, 12
Mathematical modeling languages Formal Language Theory Monday, November 19, 12
Mathematical modeling languages Formal Language Theory ֶ Monday, November 19,
12
Mathematical modeling languages Formal Language Theory ֶ 语จ Monday, November
19, 12
Regular Expression expresses Regular Grammar which recognizes Regular Language, which
is Non-Recursive Monday, November 19, 12
Parsing Expression Grammar recognizes parsing expression language, which can be
Recursive Monday, November 19, 12
Monday, November 19, 12
Monday, November 19, 12
Monday, November 19, 12
Example -- Match the following strings: Monday, November 19, 12
զಓ Example -- Match the following strings: Monday, November 19,
12
զಓ զಓ㟬ಓզಓ Example -- Match the following strings: Monday, November
19, 12
զಓ զಓ㟬ಓզಓ զಓ㟬ಓզಓ㟬ಓզಓ Example -- Match the following strings: Monday,
November 19, 12
զಓ զಓ㟬ಓզಓ զಓ㟬ಓզಓ㟬ಓզಓ զಓ㟬ಓզಓ㟬ಓզಓ㟬ಓզಓ Example -- Match the following strings:
Monday, November 19, 12
զಓ զಓ㟬ಓզಓ զಓ㟬ಓզಓ㟬ಓզಓ զಓ㟬ಓզಓ㟬ಓզಓ㟬ಓզಓ ... Example -- Match the following
strings: Monday, November 19, 12
The Language (A Regular Language): L = { զಓ, զಓ㟬ಓզಓ,
... } Monday, November 19, 12
The Regexp (A Regular Grammar): /զಓ(㟬ಓզಓ)*/ Monday, November 19, 12
Structural Analysis: զಓ(㟬ಓ( զಓ(զಓ( 㟬ಓ(զಓ) )) )) Monday, November 19,
12
Grammar in BNF Notation: <A> ::= “զಓ” <B>? <B> ::=
“㟬ಓ” <A> Monday, November 19, 12
Grammar in BNF Notation: <A> ::= “զಓ” <B>? <B> ::=
“㟬ಓ” <A> Tail Recursion Monday, November 19, 12
The “PEG-Flavored” Regexp: / (?<A> զಓ \g<B>? ){0} (?<B> 㟬ಓ
\g<A> ){0} \g<A> /x Monday, November 19, 12
ᥨښੋষ⻥鱼ത࢜తၰࢠ The really-recursive example (1): Monday, November 19, 12
ᥨښੋষ⻥鱼ത࢜తၰࢠ The really-recursive example (1): ओ语 Monday, November 19, 12
ᥨښੋষ⻥鱼ത࢜తၰࢠ The really-recursive example (1): ओ语 谓语 Monday, November 19,
12
ᥨښੋষ⻥鱼ത࢜తၰࢠ The really-recursive example (1): ओ语 宾语 谓语 Monday, November
19, 12
(?<ओ语>ᥨښ) (?<谓语>ੋ) (?<宾语>ষ⻥鱼ത࢜తၰࢠ) (?<陈ड़۟>\g<ओ语>\g<谓语>\g<宾语>) Monday, November 19, 12
ᥨښੋষ⻥鱼ത࢜తၰࢠ, Ҽ为ᥨষ⻥鱼༗ീḰ The really-recursive example (2): Monday, November 19, 12
ᥨښੋষ⻥鱼ത࢜తၰࢠ, Ҽ为ᥨষ⻥鱼༗ീḰ The really-recursive example (2): 陈ड़۟ Monday, November 19,
12
ᥨښੋষ⻥鱼ത࢜తၰࢠ, Ҽ为ᥨষ⻥鱼༗ീḰ The really-recursive example (2): 陈ड़۟ ݪҼဓ۟ Monday, November
19, 12
(?<ݪҼဓ۟>Ҽ为ᥨষ⻥鱼༗ീḰ) (?<ྫྷস话>\g<陈ड़۟>,\g<ݪҼဓ۟>) Monday, November 19, 12
զࡏRubyConfChina্讲ྃྫྷস话: “ᥨښੋষ⻥鱼ത࢜త ၰࢠ, Ҽ为ᥨষ⻥鱼༗ീḰ” The really-recursive example (3): Monday, November
19, 12
զࡏRubyConfChina্讲ྃྫྷস话: “ᥨښੋষ⻥鱼ത࢜త ၰࢠ, Ҽ为ᥨষ⻥鱼༗ീḰ” The really-recursive example (3): 陈ड़۟ Monday,
November 19, 12
զࡏRubyConfChina্讲ྃྫྷস话: “ᥨښੋষ⻥鱼ത࢜త ၰࢠ, Ҽ为ᥨষ⻥鱼༗ീḰ” The really-recursive example (3): 陈ड़۟ 宾语ಉҐ语
Monday, November 19, 12
(?<ओ语>ᥨښ|զ) (?<谓语>ੋ|ࡏRubyConfChina্讲ྃ) (?<宾语>ষ⻥鱼ത࢜తၰࢠ|ྫྷস话:\g<宾语ಉҐ语>) (?<宾语ಉҐ语>“\g<陈ड़۟>”) Monday, November 19, 12
զࡏRubyConfChina্讲ྃྫྷস话: “ᥨښੋষ⻥鱼ത࢜త ၰࢠ, Ҽ为ᥨষ⻥鱼༗ീḰ”,ୠେՈ༗স The really-recursive example (4): Monday, November
19, 12
զࡏRubyConfChina্讲ྃྫྷস话: “ᥨښੋষ⻥鱼ത࢜త ၰࢠ, Ҽ为ᥨষ⻥鱼༗ീḰ”,ୠେՈ༗স The really-recursive example (4): 转ંဓ۟ Monday,
November 19, 12
(?<转ંဓ۟>ୠେՈ༗স) (?<ྫྷস话>\g<陈ड़۟>, (\g<ݪҼဓ۟>|\g<转ંဓ۟>)) Monday, November 19, 12
Combine them all Monday, November 19, 12
/ (?<ओ语> ᥨښ|զ ){0} (?<谓语> ੋ|ࡏRubyConfChina্讲ྃ ){0} (?<宾语> ষ⻥鱼ത࢜తၰࢠ|ྫྷস话:\g<宾语ಉҐ语> ){0}
(?<陈ड़۟> \g<ओ语>\g<谓语>\g<宾语> ){0} (?<ྫྷস话> \g<陈ड़۟>,(\g<ݪҼဓ۟>|\g<转ંဓ۟>) ){0} (?<ݪҼဓ۟> Ҽ为ᥨষ⻥鱼༗ീḰ ){0} (?<宾语ಉҐ语> “\g<ྫྷস话>” ){0} (?<转ંဓ۟> ୠେՈ༗স ){0} \g<ྫྷস话> /x Monday, November 19, 12
Use dictionaries in the sentence components, you can make a
natural language parser with “Regexp” (PEG in fact) Monday, November 19, 12
Use dictionaries in the sentence components, you can make a
natural language parser with “Regexp” (PEG in fact) (?<ओ语>ᥨښ|㫴ښ|౮笼ښ|...) Monday, November 19, 12
Real world language is a bit more than PEG, generally
Context Free Grammar. Monday, November 19, 12
In CFG, the branches are not ordered: Assume A and
B are two rules, A|B is the same as B|A in CFG. Monday, November 19, 12
Even CFG can’t solve some ambiguity Monday, November 19, 12
Even CFG can’t solve some ambiguity ༗Ұେၣረਖ਼ࡏۙ Monday, November 19,
12
Even CFG can’t solve some ambiguity ༗Ұେၣረਖ਼ࡏۙ ( ) ??
Monday, November 19, 12
Now you know more than regexp parsec, rsec, treetop, parselet
... Monday, November 19, 12
Simple markdown parser in 130 lines (many features ignored but...)
Real world example Monday, November 19, 12
It supports nested parens! (while ruby-china doesn’t) [ruby](http://en.wikipedia.org/wiki/Ruby_(programming_language)) <a href=’
http://en.wikipedia.org/wiki/Ruby_(programming_language)’> ruby </a> Monday, November 19, 12
The parser for nested parens: /(?<paren> \( ( [^\(\)]+ #
non-paren chars | # or \g<paren> # a paren )* \) )/x Monday, November 19, 12
★ ri Regexp ★ https://github.com/k-takata/Onigmo/tree/master/doc/RE ★ http://en.wikipedia.org/wiki/Parsing_Expression_Grammar Helpful Links Monday,
November 19, 12
More sugars with Onigmo If there’s time... Monday, November 19,
12
/\p{Han}/ /\p{Hiragana,Katakana}/ /\g'name'/ /\g<3>/ /\g'-3'/ /\k<1>/ # /\1/ /\k'name'/ /\k<-1>/
/\k'-n-level'/ Monday, November 19, 12
/(?<=backward)something(?=forward)/ /(?<!backward)something(?!forward)/ /(?<\b)hello/ # 1.9 :( # 2.0 :) Monday,
November 19, 12
? + * +? *? ?+ ++ *+ Can you
tell all of them? Monday, November 19, 12
Thanks Monday, November 19, 12