Clojure Scraping Utils Shriphani Palakodety! spalakod@cs.cmu.edu

Scraping • Dataset construction (unstructured -> structured)! • APIs not widespread! • Unavoidable

Libraries • clj-­‐xpath  (XPath Queries)   • enlive  (CSS selectors)

XPath Queries • Select nodes in an XML document! • Clojure lib:  clj-­‐xpath  - wraps the Java XPath library

XPath Example • Select node  lr   • /root/l/lr   • /root//lr   • Infinite queries exist root l r ll lr

XPath Query Predicates • Regex matches on  a node’s  attributes  using   contains

Example //html/div[contains(@class,  ‘hello’)]/a          

Position Information • Need to refine your query sometimes.! • Examples: first child of some node

Example //html/div[2]/a          

Workflow • Parse HTML  (HtmlCleaner)   • Convert to an  org.w3c.dom.document  object   • run XPath queries on this object

Demo • Extract article metadata from reddit (title, submitter, number of comments)

CSS Selectors • Pick html elements we want to style! • Some Examples:! • Select a div with class hello:! • div.hello   • Select a div with id hello:! • div#hello   • XPath queries and CSS selectors are equivalent

Enlive • HTML document is just a clojure map                 {:tag      :html,    :attrs  nil,    :content      ({:tag      :body,          :attrs  nil,          :content          ({:tag    :div,         :attrs  nil,         :content         ("HELLO")})})}

Enlive Workflow • Process your document:! • (enlive/html-­‐resource  html-­‐stream)   • Select your elements:! • (enlive/select  processed-­‐doc  selector)

Enlive Selectors • Typically a list of nodes (not necessarily contiguous)! • Each element of this list is a keyword that represents a CSS selector

Example                 {:tag      :html,    :attrs  nil,    :content      ({:tag      :body,          :attrs  nil,          :content          ({:tag    :div,         :attrs  nil,         :content         ({:tag      :a,          :attrs  {:href                            “hello.html”},          :content          (“HELLO”)})})})} [:a]

Example II #{[div.foo  div.bar]} ({:tag      :html,      :attrs  nil,      :content      ({:tag      :body,          :attrs  nil,          :content          ({:tag      :div,                :attrs  {:class  "foo"},                :content  nil}            {:tag      :div,                :attrs  {:class  "bar"},                :content  nil})})})                        

Demo • Extract  article  metadata  from  reddit   (title,  submitter,  number  of  comments)

Best of Both Worlds • Want to use XPaths (legacy codebase)! • Also want to use enlive! • It is possible

Best of Both Worlds • (:use  [clj-­‐xpath.core  :only  [$x]])   • ($x  “//html”  html-­‐object)                 {:tag      :html,    :attrs  nil,    :content      ({:tag      :body,          :attrs  nil,          :content          ({:tag    :div,         :attrs  nil,         :content         ("HELLO")})})}

Further Reading • Enlive can be used for HTML templates and more complicated scraping: ! • https://github.com/swannodette/enlive-tutorial! • More on XPath, CSS Selectors:! • http://www.w3.org/TR/CSS2/selector.html! • Enlive-style selector syntax in the browser (using cljs):! • https://github.com/prismatic/dommy/

Questions?