Introduction to Regular Expressions

Transcript

1. Intro to Regular Expressions Presented by: Jamie Ly
2. Agenda •Definition •Infamy •Whetting •Literacy •Potpourri
3. [a-z0-9\.]+@wharton\.upenn\.edu
4. What is a Regex? • Pattern matcher • Text processor • mini-specification
5. How are they processed? • One method of processing used FSAs ◦ http://osteele.com/tools/reanimator/
6. Why the bad rap?
7. Some people, when confronted with a problem, think 'I know, I’ll use regular expressions.' Now they have two problems. -Jamie Zawinski
8. Overused Unintelligible
9. Obscure Regexes E-Mail Validation, Complete Spec http://ex-parrot.com/~pdw/Mail-RFC822-Address.html Sudoku Solver http://perl.abigail.be/Talks/Sudoku/HTML/ (direct link)
10. Whetting
11. Features
12. Feature: Literals • 1 2 3 • a b c • A B C • - # % • \. \^ \$ • \t \n \r /abc/
13. Feature: Character classes/sets • enclosed with brackets • OR • ba[rs]k matches bark and bask • Negations ^ • c[^r]ap matches anything except crap • Short-hand ◦ Perl: \d, \s, \w ◦ POSIX: [:digit:], [:space:], [:word:] /a[bc]/
14. Feature: Quantifiers • ? one or none • * >=0 • + > 0 • {n} = n • {n, m} between n and m /a[bc]+/
15. Feature: Grouping • Uses parens • For capture • Extraction • Backreferences* /(a[bc])+/
16. Feature: Grouping - Capture search = "apple pear scrapple snapple peach pineapple" re = "([a-z]+pple)" matches = RegexMatch ( search, re ) print matches ["apple", "scrapple", "snapple", "pineapple"]
17. Feature: Others • Modifiers • Start/end anchors • Zero-length matches • Backreferences • Lookahead, lookbehind, lookaround, ...
18. Examples
19. Example: Quantifiers /(not)? to be/ /((lion)+(tiger)+(bear)+)/
20. Example: Quantifiers - Matches /(not)? to be/ "not to be" "to be"
21. Example: Quantifiers - No match /(not)? to be/ "NOT to be" "not to be"
22. Example: Quantifiers - Matches /((lion)+(tiger)+(bear)+)/ "liontigerbear" "lionliontigertigerbear"
23. Example: Quantifiers - No Match /((lion)+(tiger)+(bear)+)/ "lion tiger bear" "lionbear"
24. Example: Grouping - Alternation /the (best|worst) of times/ /(liberty|death)/
25. Uses
26. 4 Main Uses •Search •Normalization •Extraction •Validation
27. Use: Search - Finding Text In Files • grep • findstr (Windows) • eclipse, Notepad++, ... • Possible Uses: ◦ files containing @todo or !todo ◦ files using cf tags
28. Use: Search - Using Regex In Code • ColdFusion - REReplace note • Javascript - RegExp • Java - java.util.regex.Pattern • ASP - VBScript.Regexp • C# - System.Text.RegularExpressions.Regex • Comparison http://en.wikipedia. org/wiki/Comparison_of_regular_expression_engines
29. Use: Search - Another Domain • Programming language detection ◦ Pastebin ◦ Syntax Highlighting
30. Use: Normalization • Legacy system data (simple flat-file processing) • One-off user lists • Phone numbers, zip code • Example: Phone Number List • Common use case: Joining/Splitting Strings • Common use case: Code refactoring
31. Use: Extraction • Ties into search and normalization • Example: URLs, e-mail addresses • Plug: CiteThis! sample • Use: Screen scraping Yahoo Finance src code
32. Use: Validation • User input • Casting v String Matching • Example: Input Masking, Stripping HTML
33. When to Avoid RE • HTML Parsing ◦ DOM/XPath • Performance critical • Stateful processing ◦ odd number of a's • Some text file processing (some CSV)
34. Regex Literacy
35. How to Read REs • Why be able to read them? • Decompose a regex into its component parts ◦ How to solve? 1+3*4/5^6+3+5-6*10+3-5 ◦ Decompose/Group! • Be familiar with the problem space/domain • Write down strings that match as you scan • Test against various strings (only use this if stumped!)
36. /^#?([a-f0-9]{6}|[a-f0-9]{3})$/
37. /^#?([a-f0-9]{6}|[a-f0-9]{3})$/
38. /^#?( [a-f0-9]{6} | [a-f0-9]{3} )$/
39. #aaa af0 af9f04 #00cc00
40. /^([a-z0-9_\.-]+)@([\da-z\.-]+)\. ([a-z\.]{2,6})$/
41. /^([a-z0-9_\.-]+)@([\da-z\.-]+)\. ([a-z\.]{2,6})$/
42. /^ ([a-z0-9_\.-]+) @ ([\da-z\.-]+) \. ([a-z\.]{2,6}) $/
43. a@-.az -@000.apples _______@0------0... jamiely@wharton.upenn.edu
44. Regex Construction
45. How to Construct a RE • Have a few examples of : ◦ matching strings ◦ not matching strings • start with a simple expression • build up • like writing pseudo-code • Let's write a date-matching RE!
46. Just because we can... does it mean we should? Casting v RegEx • Simple natural language parsing ◦ Search queries ◦ I Want Sandy ◦ Remember the milk
47. Date Match! 1. What will we match? • Typical US Dates? • International format? 2010-03-04 • Do we match times? Let's settle on matching: • Month date, year • Including 3-letter month names
48. Date Match! 1. What will we match? (continued) • Possible matches ◦ Oct. 20, 2000 • Not matches ◦ 10/20/2000 ◦ October 20
49. Date Match! 2. Start with a simple expression Write a Regex to match "October 20, 2000" /October 20, 2000/ Easy huh? Now, be able to match each day in October. /October [1-3][0-9], 2000/
50. Date Match! 2. Start with a simple expression (cont) Close! But it doesn't match the 1st through 9th /October [1-3]?[0-9], 2000/ Better! Now, match any year from 1000 to 3999. /October [1-2]?[0-9], [123][0-9][0-9][0-9]/
51. Date Match! 2. Start with a simple expression (cont) When you see the same classes repeated, you can simplify! /October [1-2]?[0-9], [123][0-9]{3}/ Now, match October's abbreviation. /(October|Oct\.) [1-2]?[0-9], [123][0-9]{3}/
52. Date Match! 2. Start with a simple expression (cont) Now we can add the other months. We'll only add May and December so we can limit the size. /((May|October|December)|(Oct|Dec)\.) [1-2]?[0-9], [123][0-9]{3} / or /(May|October|December|Oct\.|Dec\.) [1-2]?[0-9], [123][0-9]{3}/
53. Date Match! 2. Start with a simple expression (cont) From here, we determine whether to loosen the requirements. • Ignore whitespace? • Ignore case? • Comma, other punctuation optional? /(May|October|December|Oct\.?|Dec\.?)\s+[1-2]?[0-9],?\s+[123] [0-9]{3}/
54. Potpourri
55. Regex Builders • Not crutches! • Testers/Interactive builder ◦ http://rubular.com/ ◦ http://gskinner.com/RegExr/ ◦ *http://ryanswanson.com/regexp/#start ◦ http://osteele.com/tools/rework/# ◦ http://txt2re.com/index-php.php3 ◦ http://tools.netshiftmedia.com/regexlibrary/
56. Custom Project Using REs Word Jumble! http://scorpio-dev.wharton.upenn.edu/users/jamiely/wordjumble/
57. http://www.addedbytes.com/cheat-sheets/regular-expressions-cheat-sheet/
58. Appendices
59. References • http://net.tutsplus.com/tutorials/other/8-regular-expressions-you-should-know/ • http://regexlib.com/ • http://www.regular-expressions.info/
60. Unused slides
61. RE Linksheet
62. Feature: Backreference • You may use a captured group in a regex • This is useful for paired data such as html • When you don't know what the first match will be Matches an html tag and its closing tag (kinda) /<(\w+).+> inner html including tags </\1>/ Matches a line of the name song! /\w(\w+), B\1, Bo B\1, Banana Fanana/
63. Feature: Modifiers • (g)lobal • (i)gnore case/case (i)nsensitive • (m)ultiline • example
64. Feature: Start/end anchors • ^ begin • $ end • Maria: Let's start at the very beginning! • ^do re mi • REM: world as we know it$
65. Example: Quantifiers - Alt. Regex /((lion)+(tiger)+(bear)+)/ /((lion){1,2}(tiger){1,2}(bear)?)/ "liontigerbear" "lionliontigertigerbear"
66. Example: Grouping - Matches /(circle of (hell|the (inferno|abyss))){1,7}/ "circle of the inferno" "circle of hellcircle of hell"
67. Example: Grouping - No match /(circle of (hell|the (inferno|abyss))){1,7}/ "circle of abyss" "circle of"
68. *Feature: - Zero Length Matches • Zero-length matches (^, \b)
69. Use: Capture /(heart|mind)*/
70. Search • dir/ls • Example: Find files beginning with test • find • Example: Find files beginning with test and ending with a numeric timestamp. • Common use case: ◦ finding all log files ◦ finding image files
71. Use: Normalization (cont) • Phone numbers, zip code • Example: Phone Number List • Common use case: Joining/Splitting Strings
72. General Search • IDE ◦ Notepad++ ▪ sql search ◦ Eclipse ▪ function search ◦ Dreamweaver? ◦ Word? ▪ Not really regex ◦ Emacs? • FindStr/Grep ◦ Example: ? ◦ Common use case: Finding all instances of a global variable
73. Search • find
74. grep, findstr
75. RE Flavors and Compensating How to compensate for a deficient implementation. (ColdFusion)
76. What is a regular expression? • formal language • interpreted by RE processor • mini-programs • mini-specifications • domain: text-processing • iffy: find:RE::dom:XPath ... kinda • show program parse tree? ◦ compare to regex automata graph ◦ graph: ca[tp]er against cater caper acapella
77. Disperse Examples and Tests Throughout? • Roman numerals: http://stackoverflow. com/questions/800813/what-is-the-most-difficult- challenging-regular-expression-you-have-ever- written/800932 •
78. RE Cheatsheet • How to encourage RE use? • Distribute cheatsheets • Comb through LL source for examples where REs could be used?
79. Use: Search Domains • Find all files containing @todo or !todo • Find all files using cf tags