Markovian Opinions

Probabilistic generation of comment-section-worthy opinions

Transcript

1. Markovian Opinions Tom O’Brien
2. Markov Chains
3. Memoryless State Machines Fixed probability of moving from one state to the next. The evolution of the system depends only on its current state.
4. Basically Everywhere Web Pages (PageRank) Genetics Boardgames Text Prediction and Generation
6. Generating Text
7. n-grams A sequence of n words in a sentence. (defn n-grams [n s] (partition n 1 (clojure.string/split s #"\s+"))) ! (n-grams 2 "The quick brown fox") => (("The" "quick") ("quick" "brown") ("brown" "fox"))
8. (defn next-states [unigrams bigrams term] (->> (for [[[_ next] y] (bigram-term-frequencies bigrams term) :when y] [next (/ y (unigram-term-frequencies unigrams term))]) (sort-by last)))
9. (defn choose-at [p [[term q] & rest]] (when q (if (and q (< p q)) term (recur (- p q) rest)))) ! (defn next-term [unigrams bigrams term] (choose-at (rand) (next-states unigrams bigrams term))) ! (defn markov-text [unigrams bigrams initial-term] (iterate (partial next-term unigrams bigrams) initial-term))
10. Twitter Scraping
11. Harvesting Links Building a web crawler: too ambitious. (defn dailym-dot-ai-url? [url] (if url (= "dailym.ai" (.getHost (URI. url))) false)) ! ;; timeline is lazy, built with twitter-api (defn article-urls [screen-name] (filter dailym-dot-ai-url? (map expanded-url (remove retweet? (timeline screen-name)))))
12. Scraping with Hickory (require '[hickory.select :as s]) ! (defn url->hickory-response [url & {:as params}] (as-hickory (parse (:body (http/get url params))))) ! (defn article-url->comments [article-url] (map (comp parse-text :content) (s/select (s/child (s/class :comment-text)) (url->hickory-response article-url))))
13. Generating Tweets
14. Opinionate! http://twitter.com/mailonline_bot (loop [] (let [message (make-tweet (rand-nth screen-names))] (println (str "Tweeting: " message)) (update-status! mailonline-oauth-creds message) (Thread/sleep 30000) (recur)))
15. Thanks! Twitter: @_tobrien / @mailonline_bot GitHub: t-ob/markov