A mail filter application in Ursala

Presented during a hacker meetup at Songkick headquarters in London on April 18, 2008, this presentation takes a walk through some code to show how Ursala thinks.

Transcript

1. A mail filter application in Ursala Dennis Furey, Ph.D. Faculty of Business, Computing, and Information Management London South Bank University April 18, 2008 Dennis Furey, Ph.D. A mail filter application in Ursala
2. A hacker’s functional language Skip the hype. Let’s see some code. email messages in a file with mbox format some junk mail contains urls in the message body disposable host names persistent ip addresses delete messages from the file by this criterion extract host names from urls look up ip addresses match against a known blacklist Dennis Furey, Ph.D. A mail filter application in Ursala
3. Overview of the app 50 lines of mostly fluff in 3 sections . . . front matter #import std #import nat #import cli usage = -[ blip - block mail containing urls with blacklisted ip addresses Copyright (C) 2008 Dennis Furey usage: blip BLACKLIST MBOXFILE file parameters: BLACKLIST contains one dotted decimal ip address per line MBOXFILE contains mail messages in standard mbox format]- . . . Dennis Furey, Ph.D. A mail filter application in Ursala
4. Overview of the app 50 lines of mostly fluff in 3 sections . . . user defined utilities ------------------- function declarations ------------------------ messages = # lexes a mailbox file into a list of messages rlc ~&r; not =]’From ’&& sep‘ ; ~&F; length==7&& ~&w/‘@+ ~&th urls = # extracts a list of urls from a message (takewhile ~<’<"> ’)*F+ * ->~&t ~&i&& not =]’http://’ host_names = # extracts host names a list of urls (~&w/‘.)*~+ ~&s+ * skip/7; takewhile -=letters--digits--’.’ ip_addresses = # maps a list of host names to their ip addresses *= skip/*9+ =]’Address: ’*~+ ~&hm+ ask(bash)/<>+ <.’nslookup ’--> blockable = # takes (blacklist,message) and returns a boolean ^|c/~&F ip_addresses+ host_names+ urls Dennis Furey, Ph.D. A mail filter application in Ursala
5. Overview of the app 50 lines of mostly fluff in 3 sections . . . . . . driver ------------------------ main program ----------------------------- #comment usage #executable <’par’,’’> #optimize+ blip = ~command.files; ~&itBttPZB?\usage!% ~&iNC+ file$[ contents: ~contents*; (not blockable)*~rSL^D/~&h messages+~&th, path: ~&th.path, stamp: &!] Dennis Furey, Ph.D. A mail filter application in Ursala
6. Overview of the app and a black list input file like this blacklist 139.91.201.35 217.74.65.162 220.113.33.113 220.113.33.40 61.188.87.42 61.188.87.54 205.188.96.215 205.188.98.215 205.188.224.215 205.188.97.215 205.188.99.215 89.187.46.24 Dennis Furey, Ph.D. A mail filter application in Ursala
7. Walk through the messages function Taking a list of strings from the lines of a mailbox file, split it into a list of lists of strings. rlc ~&r; not =]’From ’&& sep‘ ; ~&F; length==7&& ~&w/‘@+ ~&th Dennis Furey, Ph.D. A mail filter application in Ursala
8. Walk through the messages function Taking a list of strings from the lines of a mailbox file, split it into a list of lists of strings. rlc ~&r; not =]’From ’&& sep‘ ; ~&F; length==7&& ~&w/‘@+ ~&th infer that consecutive lines belong to the same message iff Dennis Furey, Ph.D. A mail filter application in Ursala
9. Walk through the messages function Taking a list of strings from the lines of a mailbox file, split it into a list of lists of strings. rlc ~&r; not =]’From ’&& sep‘ ; ~&F; length==7&& ~&w/‘@+ ~&th infer that consecutive lines belong to the same message iff the latter line does not Dennis Furey, Ph.D. A mail filter application in Ursala
10. Walk through the messages function Taking a list of strings from the lines of a mailbox file, split it into a list of lists of strings. rlc ~&r; not =]’From ’&& sep‘ ; ~&F; length==7&& ~&w/‘@+ ~&th infer that consecutive lines belong to the same message iff the latter line does not begin with the word ’From’ and Dennis Furey, Ph.D. A mail filter application in Ursala
11. Walk through the messages function Taking a list of strings from the lines of a mailbox file, split it into a list of lists of strings. rlc ~&r; not =]’From ’&& sep‘ ; ~&F; length==7&& ~&w/‘@+ ~&th infer that consecutive lines belong to the same message iff the latter line does not begin with the word ’From’ and contain seven non-empty space delimited fields Dennis Furey, Ph.D. A mail filter application in Ursala
12. Walk through the messages function Taking a list of strings from the lines of a mailbox file, split it into a list of lists of strings. rlc ~&r; not =]’From ’&& sep‘ ; ~&F; length==7&& ~&w/‘@+ ~&th infer that consecutive lines belong to the same message iff the latter line does not begin with the word ’From’ and contain seven non-empty space delimited fields of which the second contains the @ character Dennis Furey, Ph.D. A mail filter application in Ursala
13. Walk through the urls function Taking a list of strings in a mail message, extract the urls. (takewhile ~<’<"> ’)*F+ * ->~&t ~&i&& not =]’http://’ Dennis Furey, Ph.D. A mail filter application in Ursala
14. Walk through the urls function Taking a list of strings in a mail message, extract the urls. (takewhile ~<’<"> ’)*F+ * ->~&t ~&i&& not =]’http://’ for each line in the message Dennis Furey, Ph.D. A mail filter application in Ursala
15. Walk through the urls function Taking a list of strings in a mail message, extract the urls. (takewhile ~<’<"> ’)*F+ * ->~&t ~&i&& not =]’http://’ for each line in the message take the tail while this is true Dennis Furey, Ph.D. A mail filter application in Ursala
16. Walk through the urls function Taking a list of strings in a mail message, extract the urls. (takewhile ~<’<"> ’)*F+ * ->~&t ~&i&& not =]’http://’ for each line in the message take the tail while this is true the line is non-empty and Dennis Furey, Ph.D. A mail filter application in Ursala
17. Walk through the urls function Taking a list of strings in a mail message, extract the urls. (takewhile ~<’<"> ’)*F+ * ->~&t ~&i&& not =]’http://’ for each line in the message take the tail while this is true the line is non-empty and it does not start witn ’http://’ Dennis Furey, Ph.D. A mail filter application in Ursala
18. Walk through the urls function Taking a list of strings in a mail message, extract the urls. (takewhile ~<’<"> ’)*F+ * ->~&t ~&i&& not =]’http://’ for each line in the message take the tail while this is true the line is non-empty and it does not start witn ’http://’ for each remaining non-empty line Dennis Furey, Ph.D. A mail filter application in Ursala
19. Walk through the urls function Taking a list of strings in a mail message, extract the urls. (takewhile ~<’<"> ’)*F+ * ->~&t ~&i&& not =]’http://’ for each line in the message take the tail while this is true the line is non-empty and it does not start witn ’http://’ for each remaining non-empty line accumulate the maximum prefix Dennis Furey, Ph.D. A mail filter application in Ursala
20. Walk through the urls function Taking a list of strings in a mail message, extract the urls. (takewhile ~<’<"> ’)*F+ * ->~&t ~&i&& not =]’http://’ for each line in the message take the tail while this is true the line is non-empty and it does not start witn ’http://’ for each remaining non-empty line accumulate the maximum prefix not containing angle brackets, double quotes, or spaces Dennis Furey, Ph.D. A mail filter application in Ursala
21. Walk through the host names function Taking a list of strings representing urls, extract the host names. (~&w/‘.)*~+ ~&s+ * skip/7; takewhile -=letters--digits--’.’ Dennis Furey, Ph.D. A mail filter application in Ursala
22. Walk through the host names function Taking a list of strings representing urls, extract the host names. (~&w/‘.)*~+ ~&s+ * skip/7; takewhile -=letters--digits--’.’ for each url Dennis Furey, Ph.D. A mail filter application in Ursala
23. Walk through the host names function Taking a list of strings representing urls, extract the host names. (~&w/‘.)*~+ ~&s+ * skip/7; takewhile -=letters--digits--’.’ for each url delete the seven character ’http://’ prefix Dennis Furey, Ph.D. A mail filter application in Ursala
24. Walk through the host names function Taking a list of strings representing urls, extract the host names. (~&w/‘.)*~+ ~&s+ * skip/7; takewhile -=letters--digits--’.’ for each url delete the seven character ’http://’ prefix accumulate the maximal remaining prefix Dennis Furey, Ph.D. A mail filter application in Ursala
25. Walk through the host names function Taking a list of strings representing urls, extract the host names. (~&w/‘.)*~+ ~&s+ * skip/7; takewhile -=letters--digits--’.’ for each url delete the seven character ’http://’ prefix accumulate the maximal remaining prefix containing only letters, digits, and periods Dennis Furey, Ph.D. A mail filter application in Ursala
26. Walk through the host names function Taking a list of strings representing urls, extract the host names. (~&w/‘.)*~+ ~&s+ * skip/7; takewhile -=letters--digits--’.’ for each url delete the seven character ’http://’ prefix accumulate the maximal remaining prefix containing only letters, digits, and periods eliminate duplicate host names Dennis Furey, Ph.D. A mail filter application in Ursala
27. Walk through the host names function Taking a list of strings representing urls, extract the host names. (~&w/‘.)*~+ ~&s+ * skip/7; takewhile -=letters--digits--’.’ for each url delete the seven character ’http://’ prefix accumulate the maximal remaining prefix containing only letters, digits, and periods eliminate duplicate host names retain only those containing at least one period Dennis Furey, Ph.D. A mail filter application in Ursala
28. Walk through the ip addresses function Find the ip address of each host name in a list. *= skip/*9+ =]’Address: ’*~+ ~&hm+ ask(bash)/<>+ <.’nslookup ’--> Dennis Furey, Ph.D. A mail filter application in Ursala
29. Walk through the ip addresses function Find the ip address of each host name in a list. *= skip/*9+ =]’Address: ’*~+ ~&hm+ ask(bash)/<>+ <.’nslookup ’--> for each host name Dennis Furey, Ph.D. A mail filter application in Ursala
30. Walk through the ip addresses function Find the ip address of each host name in a list. *= skip/*9+ =]’Address: ’*~+ ~&hm+ ask(bash)/<>+ <.’nslookup ’--> for each host name construct an nslookup shell command Dennis Furey, Ph.D. A mail filter application in Ursala
31. Walk through the ip addresses function Find the ip address of each host name in a list. *= skip/*9+ =]’Address: ’*~+ ~&hm+ ask(bash)/<>+ <.’nslookup ’--> for each host name construct an nslookup shell command launch it in a bash session Dennis Furey, Ph.D. A mail filter application in Ursala
32. Walk through the ip addresses function Find the ip address of each host name in a list. *= skip/*9+ =]’Address: ’*~+ ~&hm+ ask(bash)/<>+ <.’nslookup ’--> for each host name construct an nslookup shell command launch it in a bash session filter the address lines from the output transcript Dennis Furey, Ph.D. A mail filter application in Ursala
33. Walk through the ip addresses function Find the ip address of each host name in a list. *= skip/*9+ =]’Address: ’*~+ ~&hm+ ask(bash)/<>+ <.’nslookup ’--> for each host name construct an nslookup shell command launch it in a bash session filter the address lines from the output transcript delete the nine character ’Address: ’ prefixes Dennis Furey, Ph.D. A mail filter application in Ursala
34. Walk through the blockable predicate Given (blacklist,message), detect links to blacklisted addresses. ˆ |c/~&F ip_addresses+ host_names+ urls Dennis Furey, Ph.D. A mail filter application in Ursala
35. Walk through the blockable predicate Given (blacklist,message), detect links to blacklisted addresses. ˆ |c/~&F ip_addresses+ host_names+ urls take urls from the message Dennis Furey, Ph.D. A mail filter application in Ursala
36. Walk through the blockable predicate Given (blacklist,message), detect links to blacklisted addresses. ˆ |c/~&F ip_addresses+ host_names+ urls take urls from the message take host names from the urls Dennis Furey, Ph.D. A mail filter application in Ursala
37. Walk through the blockable predicate Given (blacklist,message), detect links to blacklisted addresses. ˆ |c/~&F ip_addresses+ host_names+ urls take urls from the message take host names from the urls get ip addresses from the host names Dennis Furey, Ph.D. A mail filter application in Ursala
38. Walk through the blockable predicate Given (blacklist,message), detect links to blacklisted addresses. ˆ |c/~&F ip_addresses+ host_names+ urls take urls from the message take host names from the urls get ip addresses from the host names take the non-empty black list items Dennis Furey, Ph.D. A mail filter application in Ursala
39. Walk through the blockable predicate Given (blacklist,message), detect links to blacklisted addresses. ˆ |c/~&F ip_addresses+ host_names+ urls take urls from the message take host names from the urls get ip addresses from the host names take the non-empty black list items test whether the intersection is non-empty Dennis Furey, Ph.D. A mail filter application in Ursala
40. Walk through the main program Given the list of input files, construct the contents of the filtered output mailbox file. ~contents*; (not blockable)*~rSLˆD/~&h messages+~&th Dennis Furey, Ph.D. A mail filter application in Ursala
41. Walk through the main program Given the list of input files, construct the contents of the filtered output mailbox file. ~contents*; (not blockable)*~rSLˆD/~&h messages+~&th take the contents of the input files Dennis Furey, Ph.D. A mail filter application in Ursala
42. Walk through the main program Given the list of input files, construct the contents of the filtered output mailbox file. ~contents*; (not blockable)*~rSLˆD/~&h messages+~&th take the contents of the input files split the mailbox file contents into messages Dennis Furey, Ph.D. A mail filter application in Ursala
43. Walk through the main program Given the list of input files, construct the contents of the filtered output mailbox file. ~contents*; (not blockable)*~rSLˆD/~&h messages+~&th take the contents of the input files split the mailbox file contents into messages put a copy of the black list together with each message Dennis Furey, Ph.D. A mail filter application in Ursala
44. Walk through the main program Given the list of input files, construct the contents of the filtered output mailbox file. ~contents*; (not blockable)*~rSLˆD/~&h messages+~&th take the contents of the input files split the mailbox file contents into messages put a copy of the black list together with each message filter out the non-blockable pairs Dennis Furey, Ph.D. A mail filter application in Ursala
45. Walk through the main program Given the list of input files, construct the contents of the filtered output mailbox file. ~contents*; (not blockable)*~rSLˆD/~&h messages+~&th take the contents of the input files split the mailbox file contents into messages put a copy of the black list together with each message filter out the non-blockable pairs take the messages out of the remaining pairs Dennis Furey, Ph.D. A mail filter application in Ursala
46. Walk through the main program Given the list of input files, construct the contents of the filtered output mailbox file. ~contents*; (not blockable)*~rSLˆD/~&h messages+~&th take the contents of the input files split the mailbox file contents into messages put a copy of the black list together with each message filter out the non-blockable pairs take the messages out of the remaining pairs flatten the list of messages into a list of strings Dennis Furey, Ph.D. A mail filter application in Ursala
47. Discussion Some closing remarks Unreadable code? write a paragraph of comments for each line of code that makes it more readable and still shorter than C++ No one will? see Knuth’s essays on literate programming should languages force programmers to do things? More info on http://www.basis.uklinux.net/ursala or Google for Ursala. Dennis Furey, Ph.D. A mail filter application in Ursala