Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
How to scrape web contents in Clojure
Search
Sponsored
·
SiteGround - Reliable hosting with speed, security, and support you can count on.
→
ayato
January 09, 2016
Programming
120
2
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
How to scrape web contents in Clojure
ayato
January 09, 2016
Transcript
9 slides
)PXUPTDSBQF XFCDPOUFOUT JO$MPKVSF !@BZBUP@Q
͋ͽʔ $MPKVSJBO $ZCP[V4UBSUVQT *OD
8IBUJTXFCTDSBQJOH ΣϒεΫϨΠϐϯά 8FCTDSBQJOH ͱɺ ΣϒαΠτ͔ΒใΛநग़͢Δ ίϯϐϡʔλιϑτΣΞٕज़ͷ͜ͱɻ CZXJLJQFEJB
1SPCMFNT 8FCίϯςϯπߏʹ͍ۙܗΛ͍ͯ͠Δ ࣅ͍ͯΔϖʔδ͕ࢁ͋Δ͕ඍົʹҧ͏ ߏΛ୧Δ࠶ؼతͳίʔυΛॻ͘ඞཁ͕͋Δ ͍͍ͩͨ໘͍͘͞
4LZTDSBQFS ߏΛ࠶ؼతʹ୧ͬͯ͘ΕΔ ϖʔδͷλΠϓຖʹॲཧํ๏͚ͩॻ͚͍͍ ԆγʔέϯεΛฦͯ͘͠ΕΔ Ωϟογϡػߏ͕͍͍ͭͯΔ εΫϨΠϐϯά෦&OMJWFґଘ IUUQTHJUIVCDPNOBUIFMMTLZTDSBQFS
(defn seed [username from until] (let [url (str "http://twilog.org/" username)]
[{:username username :from from :until until :url url :processor ::user-page}])) (s/defprocessor user-page :cache-template "twilog/:username" :process-fn (fn [res {:keys [username]}] (let [not-registered (seq (html/select res [:div.box-info.box-icon])) not-found (seq (html/select res [:div.box-attention.box-icon]))] (cond not-registered [{:msg "This account was not registered."}] not-found [{:msg "This account was not found."}] :else [{:url (str "http://twilog.org/" username "/archives") :processor ::archives-page}])))) &YBNQMF
(defn scrape [username & [{:as options :keys [html-cache processed-cache from
until] :or {html-cache true processed-cache true from "00000000" until "99999999"}}]] (let [handler (create-handler identity options)] (handler (s/scrape (seed username from until) :html-cache html-cache :processed-cache processed-cache)))) &YBNQMF
$PODMVTJPO 4LZTDSBQFSΛ͏ͱͤʹͳΕΔ $MPKVSF࠷ߴʂ
Enjoy Clojure
More Decks by ayato
See All by ayato
マイクロサービス内で動くAPIをF#で書いている
ayato0211
1
1.8k
Clojureという言語が私逹にもたらしたもの
ayato0211
6
3.2k
3年間考え続けてきたWebアプリケーションにおけるテストの話
ayato0211
3
330
Re:REPL-Driven Development
ayato0211
3
1.4k
Meta Template Engine
ayato0211
2
1.2k
超変換! Hiccup data structure!!
ayato0211
2
700
About Integrant
ayato0211
0
620
Muscle Assert
ayato0211
0
340
Clojureを用いたWebアプリケーション開発
ayato0211
2
3.3k
Other Decks in Programming
See All in Programming
私なりのハーネスエンジニアリング ~独自の言語処理系の開発を通して~
seosoft
0
220
はじめてのPLATEAU
nokonoko1203
1
660
Spec-Driven Development - Why Vaadin Is the Perfect Fit
simas
PRO
0
260
地方こそサーバーレス — 10年続けた、顔が見えるチームで確かな基盤を / Serverless Matters Most in Regional Areas — 10 Years of Building Reliable Foundations with Close-Knit Teams
seike460
PRO
2
490
AI時代のTerminal環境を UNIX哲学で設計・運用する
yasunori0418
1
360
AI-Powered Domain-Driven Design
ziobrando
0
470
あなたの await、OS まで届いてる? strace で覗く JavaScript の非同期
hasutosasaki
1
2.4k
手の動かし方がわからなかった私が個人開発者になるまで
usagi0920_pochi
0
270
baseline으로 마이그레이션, 리팩토링 안전하게 맡기기
fornewid
1
1k
Vapor Mode でアクセシビリティは 壊れないか検証した話
fuqda
1
350
AI駆動開発4年目の現在地(『あなたはFDEになるべきか?』イベント資料)
kumaiu
0
660
[DroidKnights 2026] Workflow AX 적용기 — 자동 정책 정리부터 E2E까지
dayeon
0
110
Featured
See All Featured
B2B Lead Gen: Tactics, Traps & Triumph
marketingsoph
1
300
SEO for Brand Visibility & Recognition
aleyda
0
4.9k
Imperfection Machines: The Place of Print at Facebook
scottboms
270
14k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
133
20k
Facilitating Awesome Meetings
lara
57
7.2k
Unsuck your backbone
ammeep
672
58k
AI Search: Where Are We & What Can We Do About It?
aleyda
0
8.2k
Game over? The fight for quality and originality in the time of robots
wayneb77
1
330
Into the Great Unknown - MozCon
thekraken
41
2.8k
How To Speak Unicorn (iThemes Webinar)
marktimemedia
1
640
How People are Using Generative and Agentic AI to Supercharge Their Products, Projects, Services and Value Streams Today
helenjbeal
2
400
Why Mistakes Are the Best Teachers: Turning Failure into a Pathway for Growth
auna
0
360