EuroClojure 2012: Introduction to Cascalog

May 31, 2012

640

EuroClojure 2012: Introduction to Cascalog

Cascalog is a data processing library for Clojure. Cascalog mixes functional and logic programming. It makes data processing code very concise, easier to grasp and reason about.

Being just a Clojure library, all of Clojures’ features are just a keystroke away – no need to learn another custom language (like Pig or Hive). This also allows for powerful abstraction and composition capabilities.

This is the slide deck of my presentation at EuroClojure 2012 (http://euroclojure.com/2012). The slides are mainly about my motivations for using Cascalog. The library itself was presented in a live coding environment. As soon as the recording is available, I'll add a link to it here.

Keywords: Hadoop, Clojure, Cascalog, Pig, Hive, Logic Programming

Stefan Hübner

May 31, 2012

Tweet

More Decks by Stefan Hübner

See All by Stefan Hübner

Berlin Buzzwords 2012: Introduction to Cascalog

1

570

Introduction to Cascalog

1

170

Other Decks in Programming

See All in Programming

Semantic search with Django and pgvector

0

240

4

890

1BRC--Nerd Sniping the Java Community

0

340

GitHub Copilotのススメ

0

190

Blue/Greenデプロイの導入による運用フローの改善

1

360

GitHub Actionsで泣かないためにやっておきたい設定 / Recommended GHA settings to avoid crying

3

520

二郎系ラーメンのコールで学ぶ AST 解析

7

1.7k

Prepare for Jakarta EE 11 - Performance and Developer Productivity

0

680

Front-end application development, Symfony-style(s)

2

2k

デフォルトにして至高、RubyMineの大好きな所

0

230

スクラムガイドのスプリントレトロスペクティブを改めて読みかえしてみた / Re-reading the Sprint Retrospective Section in the Scrum Guide

3

410

チーム力を高めるスクラム実践法：カンバン公開と課題攻略について - ニフティのスクラムトーク Vol. 2 - NIFTY Tech Talk #18

1

110

Featured

See All Featured

Atom: Resistance is Futile

258

25k

実際に使うSQLの書き方徹底解説 / pgcon21j-tutorial

119

39k

Facilitating Awesome Meetings

41

5.6k

Intergalactic Javascript Robots from Outer Space

266

26k

Design by the Numbers

274

18k

Statistics for Hackers

789

220k

Designing Dashboards & Data Visualisations in Web Apps

226

51k

Why Our Code Smells

331

56k

StorybookのUI Testing Handbookを読んだ

12

4.6k

Making Projects Easy

108

5.5k

Designing for humans not robots

248

25k

VelocityConf: Rendering Performance Case Studies

320

23k

Transcript

Introduction to Cascalog Stefan Hübner, Nokia Berlin EuroClojure 2012
Nokia Maps • Address & POI Search • Category Search
• Nearby Recommendations
Hadoop • Batch Processing • (Very) Large Scale • Distributed
Filesystem • Parallel Computation • Fault-Tolerant
Hadoop • Batch Processing • (Very) Large Scale • Distributed
Filesystem • Parallel Computation • Fault-Tolerant
Hadoop • Batch Processing • (Very) Large Scale • Distributed
Filesystem • Parallel Computation • Fault-Tolerant
Hadoop MapReduce API • Tedious and verbose • Hard to
test • Hard to refactor
• Tedious and verbose • Hard to test • Hard
to refactor Hadoop MapReduce API
Pig and Hive • Deﬁne their own query language •
Custom operations in Java, Python, ... • Non-intuitive integration
(())
Star Trek I - "The Motion Picture", Paramount Pictures V'Gr's
question
Star Trek I - "The Motion Picture", Paramount Pictures "Ist
das wirklich alles? Ist da sonst gar nichts mehr?"
Cascalog Cascalog Cascading Hadoop Abstraction Variables and logic Tuples, data
workﬂows Key/value pairs, simple aggregation slide (c) Nathan Marz, reproduced with permission
Queries (<- ; deﬁnes a query [?person] ; output variables
(age ?person ?age) ; generator with two variables (< ?age 30)) ; ﬁlter
Queries (<- [?person] (age ?person ?age) ; generator with two
variables (< ?age 30)) ; ﬁlter Predicates
Predicates • Functions • Filters • Aggregators • Generators
None
Thank you! Stefan Hübner, @sthuebner http://knowyourmeme.com/memes/cereal-guy