Introduction to Regular Expressions by Luke Sneeringer

Transcript

1. Regular Expressions Luke Sneeringer @lukesneeringer
2. Why Regex? • Regular expressions are a standard mechanism to find a pattern of text within a bigger block of text. • Also useful for ensuring formatting consistency, or transforming data into a common format. @lukesneeringer
4. \b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b Dragons? @lukesneeringer
8. Learning Regex • Regular expressions actually aren't that scary. • But starting with the most complicated regular expression imaginable is. :-) @lukesneeringer
9. I The "re" module
10. The re module • Regular expression parsing is done through Python's re module. • import re
 re.search(needle, haystack) • PSA: Don't use re.match, at least while you're learning. Use re.search exclusively. @lukesneeringer
11. The re module • If your regex matches, re.search will return a "match object". • Match objects have a few methods you commonly use: group, groups, groupdict. • If your regex doesn't match, re.search will return None. @lukesneeringer
12. Python "raw" strings • In Python, you can prefix a string literal with r to get a "raw" string. • This is useful for regular expressions, as we'll see later, because it makes escaping regexes more sane. @lukesneeringer
13. II Building Blocks
14. The Basics The most basic regular expression is just a string of uninteresting characters (Latin letters, Arabic numerals). >>> re.search('abc', 'abcdef')
 <_sre.SRE_Match object at 0x100790cc8> >>> re.search('xyz', 'abcdef') >>> @lukesneeringer
15. The Basics • This is simple enough. • But regular expressions are about matching patterns, not exact text. @lukesneeringer
16. Repetition Certain characters indicate that a character is repeated a number of times. + Used one or more times * Used zero or more times ? Used zero or one times (e.g. optional) {x} Used x times (e.g. {2}) {x,y} Used between x and y times, inclusive @lukesneeringer
17. Repetition • An important note: repetition applies to the single "atomic thing" it folllows: ab{2} matches abb, not abab. • So far, the only "atomic thing" we've seen is a single literal character. • We're about to look at some more. @lukesneeringer
18. Character Sets Characters enclosed in [] characters are called character sets. This matches any single character in the set. >>> re.search('gr[ae]y', 'gray')
 <_sre.SRE_Match object at 0x100790780> >>> re.search('gr[ae]y', 'grey')
 <_sre.SRE_Match object at 0x100790cc8> >>> re.search('gr[ae]y', 'grzy') >>> @lukesneeringer
19. Character Sets Important note: Character sets match only a single occurrence of a character. gr[ae]y gray grey gry grzy graay graey Gray grAy @lukesneeringer
20. Character Sets The hyphen character with a character set is special. It designates a range of characters (usually). >>> re.search('[0-9]{3}-[0-9]{4}',
 '867-5309')
 <_sre.SRE_Match object at 0x1020b0d30> @lukesneeringer
21. Character Sets • A nice trick: If you want a literal hyphen in a character set, put it at the beginning or end
 (I prefer the end). • [a-z0-9-]+ will match one or more characters between a and z (lower-cased only), 0 and 9, and the literal hyphen character. • Obvious use case: [0-9a-f-]{36} for a Python UUID's string representation. @lukesneeringer
22. Character Classes Certain commonly used sets have character classes. These are essentially shorthands that give you common sets of characters. The expression on the previous slide could also be written thusly: [\d]{3}-[\d]{4} @lukesneeringer
23. Character Classes \w word characters
 (in Python, this includes numbers also) \d numbers \s whitespace characters
 (space, tab, newline, etc.) \b word boundary characters @lukesneeringer
24. Review This alone is sufficient for finding simple patterns. [\w ]+, [A-Z]{2} [A-Z][\d][A-Z] [\d][A-Z][\d] Montréal, QC H2Z 1X7 @lukesneeringer
25. III Intermediate Blocks
26. Negation • Character sets can also support negation, thereby matching any character that is not in the character set. • To accomplish this, use a ^ character to begin the character set. @lukesneeringer
27. Negation • Note that a negated character set does not check for the absence of a matching character. • Rather, a negated character class checks for the presence of a character that does not match a character in the set. >>> re.search(r'abc[^d]', 'abc') >>> re.search(r'abc[^d]', 'abcxyz').group()
 'abcx' @lukesneeringer
28. Special Characters • ^ matches beginning of string/line. • $ matches end of string/line. >>> re.search('abc', 'abcdef')
 <_sre.SRE_Match object at 0x100690d30> >>> re.search('^abc', 'abcdef')
 <_sre.SRE_Match object at 0x100690780> >>> re.search('^abc$', 'abcdef') >>> @lukesneeringer
29. IV Introspection
30. Introspection • So far, we've only covered the ability to match a block of text – in other words, to see whether it (or a piece of it) conforms to a pattern. • That's sometimes enough; usually, though, we want to extract text that matches a pattern and store or manipulate it. @lukesneeringer
31. Full Match Introspection • The simplest thing we might want to do is get the full match. • To do this, use the match object's group method. >>> m = re.search(r'[\d]{3}-[\d]{3}-[\d]{4}',
 r'blah 514-867-5309 blah') >>> m
 <_sre.SRE_Match object at 0x100690d30> >>> m.group()
 '514-867-5309'
32. Backreferences • Often you want to not only introspect your entire match, but also defined pieces of your match. • The tool for doing this is called backreferences. • This is a fancy name for putting subsets of your regex in parentheses. @lukesneeringer
33. Backreferences Consider the previous address example: ^[\w ]+, [A-Z]{2} [A-Z][\d][A-Z] [\d][A-Z][\d]$ Let's add backreferences: ^([\w ]+), ([A-Z]{2}) ([A-Z][\d][A-Z] [\d][A-Z][\d])$ @lukesneeringer
34. Backreferences >>> regex = r'^([\w ]+), ([A-Z]{2}) '\
 r'([A-Z][\d][A-Z] [\d][A-Z][\d])$' >>> match = re.search(regex,
 'Montréal, QC H2Z 1X7') >>> match.group()
 'Montréal, QC H2Z 1X7' >>> match.groups()
 ('Montréal', 'QC', 'H2Z 1X7') @lukesneeringer
35. Named Backreferences • Python has added an extension to Perl- compatible regular expressions to have named backreferences. • This syntax has been ported to most other languages. • The syntax is to add ?P<foo> to the beginning of the backreference. @lukesneeringer
36. Named Backreferences Our previous address with plain backreferences: ^([\w ]+), ([A-Z]{2}) ([A-Z][\d][A-Z] [\d][A-Z][\d])$ Let's make those named backreferences (apologies for the line break): ^(?P<city>[\w ]+), (?P<province>[A-Z]{2})
 (?P<postal>[A-Z][0-9][A-Z] [0-9][A-Z][0-9])$ @lukesneeringer
37. Named Backreferences >>> regex = \
 r'^(?P<city>[\w ]+), '\
 r'(?P<province>[A-Z]{2}) '\
 r'(?P<pc>[A-Z][\d][A-Z] [\d][A-Z][\d])$' >>> match = re.search(regex,
 'Montréal, QC H2Z 1X7') >>> match.group()
 'Montréal, QC H2Z 1X7' >>> match.groups()
 ('Montréal', 'QC', 'H2Z 1X7') >>> match.groupdict()
 {'city': 'Montréal', 'province': 'QC',
 'pc': 'H2Z 1X7'}
38. V The Dragon
39. \b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b The Dragon @lukesneeringer
40. VI Conclusion
41. Conclusion • There is much, much more that can be done with regular expressions. This talk has simply given the building blocks. • A few topics I haven't covered:
 alternation ("or"), lookahead, conditionals @lukesneeringer
42. Regular Expressions Luke Sneeringer @lukesneeringer