Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
Let’s write a parser! [SoundCloud HQ edition]
Denis Defreyne
May 17, 2016
Programming
0
140
Let’s write a parser! [SoundCloud HQ edition]
Denis Defreyne
May 17, 2016
Tweet
Share
More Decks by Denis Defreyne
See All by Denis Defreyne
ddfreyne
0
0
ddfreyne
0
38
ddfreyne
0
41
ddfreyne
0
89
ddfreyne
0
76
ddfreyne
0
50
ddfreyne
0
120
ddfreyne
0
110
ddfreyne
1
130
Other Decks in Programming
See All in Programming
matyo91
1
140
doyaaaaaken
1
290
coe401_
3
160
o0h
PRO
2
1.5k
masamichi
0
220
tmyk110
1
110
yusuke57
2
260
ajstarks
2
750
takuyaa
4
450
track3jyo
3
220
nerocrux
1
670
takanorihara
0
250
Featured
See All Featured
bkeepers
PRO
408
58k
keavy
107
14k
malarkey
392
61k
jasonvnalue
81
8.1k
lara
172
9.7k
morganepeng
19
1.3k
tenderlove
53
3.5k
trishagee
24
3k
iamctodd
22
2.1k
myddelton
109
11k
kneath
220
15k
tammielis
237
23k
Transcript
Let’s write a parser! DENIS DEFREYNE / SOUNDCLOUD, BERLIN /
MAY 17TH, 2016
1. Language 2
I am Denis. 3
But how do you know that I am Denis? 4
But how do you know that I am Denis? I
told you. I wrote it down. You’ve probably seen me before. Etc. 5
But how do you know that I am Denis? You
understand English. 6
Computers are stupid. 7
8 $ git commit --message="Fix bugs"
9 def greet(name) puts "Hello, #{name}" end
10 def greet(name: String): Unit = { println(s"Hello, $name!") }
Text forms a language, but computers don’t know that. 11
2. Parsing 12
Basic idea: 13 Parser objects that are small, composable, and
purely functional.
14 def read(input, pos)
15 def read(input, pos) Success.new(pos + 1) end
16 def read(input, pos) Failure.new(pos) end
17 char("H") Succeeds if the next character is the given
one.
18 char("H").apply("Hello")
18 H e l l o char("H").apply("Hello")
18 H e l l o 0 1 2 3
4 char("H").apply("Hello")
18 H e l l o 0 1 2 3
4 char("H").apply("Hello")
18 H e l l o 0 1 2 3
4 char("H").apply("Hello")
18 H e l l o 0 1 2 3
4 char("H").apply("Hello") Success(pos = 1)
19 char("H").apply("Adiós")
19 A d i ó s 0 1 2 3
4 char("H").apply("Adiós")
19 A d i ó s 0 1 2 3
4 char("H").apply("Adiós")
19 A d i ó s 0 1 2 3
4 char("H").apply("Adiós")
Failure(pos = 0) 19 A d i ó s 0
1 2 3 4 char("H").apply("Adiós")
20 if input[pos] == @char Success.new(pos + 1) else Failure.new(pos)
end
21 seq(a, b) Succeeds if both given parsers succeed in
sequence.
22 seq(char("H"), char("e")).apply("Hello")
H e l l o 22 0 1 2 3
4 seq(char("H"), char("e")).apply("Hello")
H e l l o 22 0 1 2 3
4 seq(char("H"), char("e")).apply("Hello")
H e l l o 22 0 1 2 3
4 seq(char("H"), char("e")).apply("Hello")
H e l l o 22 0 1 2 3
4 seq(char("H"), char("e")).apply("Hello")
H e l l o 22 0 1 2 3
4 seq(char("H"), char("e")).apply("Hello") Success(pos = 2)
23 seq( char("H"), char("e"), char("l"), char("l"), char("o"), )
24 string(s) Succeeds if all characters in the given string
can be read in sequence.
H e l l o 25 0 1 2 3
4 string("Hello").apply("Hello")
H e l l o 25 0 1 2 3
4 string("Hello").apply("Hello")
H e l l o 25 0 1 2 3
4 string("Hello").apply("Hello")
H e l l o 25 0 1 2 3
4 string("Hello").apply("Hello") Success(pos = 5)
26 eof() Succeeds at the end of input; fails otherwise.
H e l l o 27 0 1 2 3
4 seq(string("Hello"), eof).apply("Hello")
H e l l o 27 0 1 2 3
4 seq(string("Hello"), eof).apply("Hello")
H e l l o 27 0 1 2 3
4 seq(string("Hello"), eof).apply("Hello")
H e l l o 27 0 1 2 3
4 seq(string("Hello"), eof).apply("Hello")
H e l l o 27 0 1 2 3
4 seq(string("Hello"), eof).apply("Hello") Success(pos = 5)
28 0 1 2 3 4 5 H e l
l o ! seq(string("Hello"), eof).apply("Hello!")
28 0 1 2 3 4 5 H e l
l o ! seq(string("Hello"), eof).apply("Hello!")
28 0 1 2 3 4 5 H e l
l o ! seq(string("Hello"), eof).apply("Hello!")
28 0 1 2 3 4 5 H e l
l o ! seq(string("Hello"), eof).apply("Hello!")
28 0 1 2 3 4 5 Failure(pos = 5)
H e l l o ! seq(string("Hello"), eof).apply("Hello!")
29 alt(a, b) Succeeds if either of the given parsers
succeed.
A d i ó s 30 0 1 2 3
4 alt(char("H"), char("A")).apply("Adiós")
A d i ó s 30 0 1 2 3
4 alt(char("H"), char("A")).apply("Adiós")
A d i ó s 30 0 1 2 3
4 alt(char("H"), char("A")).apply("Adiós")
A d i ó s 30 0 1 2 3
4 alt(char("H"), char("A")).apply("Adiós") Success(pos = 1)
31 whitespace_char = alt( char(" "), char("\t"), char("\r"), char("\n"), )
32 opt(p) Succeeds always, but only advances if p succeeds.
33 repeat(p) Succeeds always, and attempts to apply p as
often as possible.
34 repeat(whitespace_char)
35 intersperse(a, b) Alternates between a and b., always ending
with a.
36 intersperse(char("a"), char(",")).apply("a,a,b")
a , a , b 36 0 1 2 3
4 intersperse(char("a"), char(",")).apply("a,a,b")
a , a , b 36 0 1 2 3
4 intersperse(char("a"), char(",")).apply("a,a,b")
a , a , b 36 0 1 2 3
4 intersperse(char("a"), char(",")).apply("a,a,b")
a , a , b 36 0 1 2 3
4 intersperse(char("a"), char(",")).apply("a,a,b")
a , a , b 36 0 1 2 3
4 intersperse(char("a"), char(",")).apply("a,a,b") Success(pos = 3)
37 etc.
3. Examples 38
39 720 6 29530
40 digit = alt( *('0'..'9') .map { |c| char(c) }
)
41 digit = char_in('0'..'9')
42 digit = char_in('0'..'9') nat_number = seq(digit, repeat(digit))
43 digit = char_in('0'..'9') nat_number = repeat1(digit)
44 digit = char_in('0'..'9') nat_number = repeat1(digit) .capture
44 digit = char_in('0'..'9') nat_number = repeat1(digit) .capture
Success(pos = 3, data = "720")
45 def read(input, pos)
46 def read(input, pos) Success.new(pos + 1) end
47 def read(input, pos) Success.new(pos + 1, "blahblah") end
48 dec_number = seq( nat_number, char('.'), nat_number, )
49 Horan,Niall,93 Payne,Liam,93 Tomlinson,Louis,91 Styles,Harry,94 Malik,Zayn,93
50 field = repeat(char_not_in(',', "\n")) line = intersperse(field, char(',')) file
= seq( line.intersperse(char("\n")), eof(), )
50 field = repeat(char_not_in(',', "\n")) line = intersperse(field, char(',')) file
= seq( line.intersperse(char("\n")), eof(), )
50 field = repeat(char_not_in(',', "\n")) line = intersperse(field, char(',')) file
= seq( line.intersperse(char("\n")), eof(), )
50 field = repeat(char_not_in(',', "\n")) line = intersperse(field, char(',')) file
= seq( line.intersperse(char("\n")), eof(), )
51 Horan,Niall,93 Payne,Liam,93 Tomlinson,Louis,91 Styles,Harry,94 Malik,Zayn,93
52 [ ["Horan", "Niall", 93], ["Payne", "Liam", 93], ["Tomlinson", "Louis",
91], ["Styles", "Harry", 94], ["Malik", "Zayn", 93], ]
53 add(1, mul(2, 3)) sub(5, 4)
54 lparen = char('(') rparen = char(')') comma = char(',')
55 expr = alt(lazy { funcall }, nat_number)
56 funcall = seq( identifier, lparen, arg_list, rparen, )
57 letter = char_in('a'..'z') identifier = repeat1(letter)
58 arg_list = intersperse( expr, seq(comma, whitespace), )
59 arg_list = opt( intersperse( expr, seq(comma, whitespace), ) )
60
60 expr_list = intersperse(expr, char("\n"))
60 expr_list = intersperse(expr, char("\n")) program = seq(expr_list, eof)
61 add(1, mul(2, 3)) sub(5, 4)
62 Success(pos = 27)
Where’s the data!!! 63
64 funcall = seq( identifier, lparen, arg_list, rparen, )
65 funcall = seq( identifier.capture, lparen, arg_list, rparen, )
66 funcall = seq( identifier.capture, lparen, arg_list, rparen, ).map do
|data| # stuff here end
67 funcall = seq( identifier.capture, lparen, arg_list, rparen, ).map do
|data| FunCall.new(data[0], data[2]) end
68 add(1, mul(2, 3)) sub(5, 4)
69 [ FunCall.new("add", [ 1, FunCall.new("mul", [2, 3]), ]), FunCall.new("sub",
[5, 4]), ]
And that is how you can write a parser. 70
And that is how you can write a parser using
parser combinators. 71
72 ḌPARSE
72 ḌPARSE A GOOD PARSER LIBRARY FOR RUBY
github.com/ddfreyne/d-parse 73
github.com/ddfreyne/d-parse 73
github.com/ddfreyne/d-parse 73
74
74 require 'd-parse'
74 require 'd-parse' module JSONGrammar
74 require 'd-parse' module JSONGrammar extend DParse::DSL
74 require 'd-parse' module JSONGrammar extend DParse::DSL DIGIT = char_in('0'..'9')
NUMBER = repeat1(DIGIT) end
74 require 'd-parse' module JSONGrammar extend DParse::DSL DIGIT = char_in('0'..'9')
NUMBER = repeat1(DIGIT) end res = Grammar::NUMBER.apply('8700')
75
75 case res
75 case res when DParse::Success puts(res.data.inspect)
75 case res when DParse::Success puts(res.data.inspect) when DParse::Failure $stderr.puts res.pretty_message
exit(1) end
76 expected identifier at line 1, column 36 def
reticulate(splines, threshold, ) { ↑
77 github.com/ddfreyne/d-parse
77 github.com/ddfreyne/d-parse PRE- ALPHA! BE AN EARLY ADOPTER!
78 My name is Denis. Ready to parse your questions.
Find me at denis@soundcloud.com, or @denis on Slack.