Clojure Scraping Overview

Clojure Scraping Utils Shriphani Palakodety! [email protected]

Scraping • Dataset construction (unstructured -> structured)! • APIs not
widespread! • Unavoidable

Libraries • clj-‐xpath (XPath Queries) • enlive (CSS selectors)

XPath Queries • Select nodes in an XML document! •
Clojure lib: clj-‐xpath - wraps the Java XPath library

XPath Example • Select node lr • /root/l/lr
• /root//lr • Inﬁnite queries exist root l r ll lr

XPath Query Predicates • Regex matches on a node’s attributes
using contains

Example //html/div[contains(@class, ‘hello’)]/a <html> <div class=‘hello’>
<a href=‘hello.html’>Hello</a> </div> <div class=‘bye’> <a href=‘bye.html’>Bye</a> </div> </html>

Position Information • Need to reﬁne your query sometimes.! •
Examples: ﬁrst child of some node

Example //html/div[2]/a <html> <div> <a
href=‘hello.html’> hello </a> </div> <div> <a href=‘hello2.html’> hello again </a> </div> </html>

Workﬂow • Parse HTML (HtmlCleaner) • Convert to an
org.w3c.dom.document object • run XPath queries on this object

Demo • Extract article metadata from reddit (title, submitter, number
of comments)

CSS Selectors • Pick html elements we want to style!
• Some Examples:! • Select a div with class hello:! • div.hello • Select a div with id hello:! • div#hello • XPath queries and CSS selectors are equivalent

Enlive • HTML document is just a clojure map <html>
<body> <div> HELLO </div> </body> </html> {:tag :html, :attrs nil, :content ({:tag :body, :attrs nil, :content ({:tag :div, :attrs nil, :content ("HELLO")})})}

Enlive Workﬂow • Process your document:! • (enlive/html-‐resource html-‐stream)
• Select your elements:! • (enlive/select processed-‐doc selector)

Enlive Selectors • Typically a list of nodes (not necessarily
contiguous)! • Each element of this list is a keyword that represents a CSS selector

Example <html> <body> <div>
<a href=‘hello.html’> HELLO </a> </div> </body> </html> {:tag :html, :attrs nil, :content ({:tag :body, :attrs nil, :content ({:tag :div, :attrs nil, :content ({:tag :a, :attrs {:href “hello.html”}, :content (“HELLO”)})})})} [:a]

Example II #{[div.foo div.bar]} ({:tag :html,
:attrs nil, :content ({:tag :body, :attrs nil, :content ({:tag :div, :attrs {:class "foo"}, :content nil} {:tag :div, :attrs {:class "bar"}, :content nil})})}) <html> <body> <div class=‘foo'> </div> <div class=‘bar'> </div> </body> </html>

Demo • Extract article metadata from reddit (title, submitter,
number of comments)

Best of Both Worlds • Want to use XPaths (legacy
codebase)! • Also want to use enlive! • It is possible

Best of Both Worlds • (:use [clj-‐xpath.core :only [$x]])
• ($x “//html” html-‐object) <html> <body> <div> HELLO </div> </body> </html> {:tag :html, :attrs nil, :content ({:tag :body, :attrs nil, :content ({:tag :div, :attrs nil, :content ("HELLO")})})}

Further Reading • Enlive can be used for HTML templates
and more complicated scraping: ! • https://github.com/swannodette/enlive-tutorial! • More on XPath, CSS Selectors:! • http://www.w3.org/TR/CSS2/selector.html! • Enlive-style selector syntax in the browser (using cljs):! • https://github.com/prismatic/dommy/

Questions?

Clojure Scraping Overview

Clojure Scraping Overview

Shriphani Palakodety

More Decks by Shriphani Palakodety

Other Decks in Programming

Featured

Transcript

Clojure Scraping Utils Shriphani Palakodety! [email protected]

Scraping • Dataset construction (unstructured -> structured)! • APIs not

Libraries • clj-‐xpath (XPath Queries) • enlive (CSS selectors)

XPath Queries • Select nodes in an XML document! •

XPath Example • Select node lr • /root/l/lr

XPath Query Predicates • Regex matches on a node’s attributes

Example //html/div[contains(@class, ‘hello’)]/a <html> <div class=‘hello’>

Position Information • Need to reﬁne your query sometimes.! •

Example //html/div[2]/a <html> <div> <a

Workﬂow • Parse HTML (HtmlCleaner) • Convert to an

Demo • Extract article metadata from reddit (title, submitter, number

CSS Selectors • Pick html elements we want to style!

Enlive • HTML document is just a clojure map <html>

Enlive Workﬂow • Process your document:! • (enlive/html-‐resource html-‐stream)

Enlive Selectors • Typically a list of nodes (not necessarily

Example <html> <body> <div>

Example II #{[div.foo div.bar]} ({:tag :html,

Demo • Extract article metadata from reddit (title, submitter,

Best of Both Worlds • Want to use XPaths (legacy

Best of Both Worlds • (:use [clj-‐xpath.core :only [$x]])

Further Reading • Enlive can be used for HTML templates

Questions?