Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cascalog

Avatar for αλεx π αλεx π
May 22, 2013
Technology
4
140

 Cascalog

Short demo talk on Cascalog on Hadoop UG in Munich

Avatar for αλεx π

αλεx π

May 22, 2013
Tweet

More Decks by αλεx π

See All by αλεx π
Scalable Time Series With Cassandra
Avatar for αλεx π ifesdjeen
1
370
Bayesian Inference is known to make machines biased
Avatar for αλεx π ifesdjeen
2
370
Cassandra for Data Analytics Backends
Avatar for αλεx π ifesdjeen
7
430
Stream Processing and Functional Programming
Avatar for αλεx π ifesdjeen
1
740
PolyConf 2015 - Rocking the Time Series boat with C, Haskell and ClojureScript
Avatar for αλεx π ifesdjeen
0
460
Clojure - A Sweetspot for Analytics
Avatar for αλεx π ifesdjeen
8
2.1k
Going Off Heap
Avatar for αλεx π ifesdjeen
3
1.9k
Always be learning
Avatar for αλεx π ifesdjeen
1
140
Learn Yourself Emacs For Great Good workshop slides
Avatar for αλεx π ifesdjeen
3
330

Other Decks in Technology

See All in Technology
Snowflake Intelligenceにはこうやって立ち向かう！クラシルが考えるAI Readyなデータ基盤と活用のためのDataOps
Avatar for harry gappy50
0
110
Webブラウザ向け動画配信プレイヤーの 大規模リプレイスから得た知見と学び
Avatar for yud0uhu yud0uhu
0
230
バッチ処理で悩むバックエンドエンジニアに捧げるAWS Glue入門
Avatar for morimorikochan diggymo
3
190
機械学習を扱うプラットフォーム開発と運用事例
Avatar for LINEヤフーTech (LY Corporation Tech) lycorptech_jp
PRO
0
230
AI開発ツールCreateがAnythingになったよ
Avatar for s sato tendasato
0
120
MCPで変わる Amebaデザインシステム「Spindle」の開発
Avatar for Ameba Spindle spindle
PRO
3
3.2k
会社紹介資料 / Sansan Company Profile
Avatar for Sansan, Inc. sansan33
PRO
6
380k
allow_retry と Arel.sql / allow_retry and Arel.sql
Avatar for Shintani Teppei euglena1215
1
160
LLMを搭載したプロダクトの品質保証の模索と学び
Avatar for SmartHR 品質保証部 qa
0
1k
ChatGPTとPlantUML/Mermaidによるソフトウェア設計
Avatar for gowhich501 gowhich501
1
130
Rustから学ぶ 非同期処理の仕組み
Avatar for skanehira skanehira
1
130
人工衛星のファームウェアをRustで書く理由
Avatar for KOBA789 koba789
13
7.3k

Featured

See All Featured
Measuring & Analyzing Core Web Vitals
Avatar for Philip Tellis bluesmoon
9
580
Rails Girls Zürich Keynote
Avatar for Gregor Martynus gr2m
95
14k
How to Think Like a Performance Engineer
Avatar for Harry Roberts csswizardry
26
1.9k
A Tale of Four Properties
Avatar for Chris Coyier chriscoyier
160
23k
jQuery: Nuts, Bolts and Bling
Avatar for Doug Neiner dougneiner
64
7.9k
Docker and Python
Avatar for Tania Allard trallard
45
3.6k
Embracing the Ebb and Flow
Avatar for Simon Collison colly
87
4.8k
The Success of Rails: Ensuring Growth for the Next 100 Years
Avatar for Eileen M. Uchitelle eileencodes
46
7.6k
BBQ
Avatar for Matthew Crist matthewcrist
89
9.8k
Intergalactic Javascript Robots from Outer Space
Avatar for Vicent Martí tanoku
272
27k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
Avatar for Kevin Liebholz kevinliebholz
34
6k
Learning to Love Humans: Emotional Interface Design
Avatar for Aarron Walter aarron
273
40k

Transcript

  1. Cascalog Hassle-free MapReduce that matches your scale Thursday, May 23,

    13

  2. Thursday, May 23, 13

  3. Setting expecations •This is not a guide •And not a

    tutorial •Doesn’t claim to be complete •Mostly to give you an idea •And encourage you to explore further Thursday, May 23, 13

  4. How much time do you spend on writing logic that

    framework should take care of? Thursday, May 23, 13

  5. How easy is it to debug your map/reduce aggragation? Thursday,

    May 23, 13

  6. Hadoop + Java composable, but too vebrose Pig, Hive too

    concrete, lack of abstraction and composition Thursday, May 23, 13

  7. Thursday, May 23, 13

  8. • Clear, declarative syntax • Inner and outer joins •

    Aggregators • Functions • Subqueries, composition • Sorting • Performant Thursday, May 23, 13

  9. Casca-WHAT? • Built on top of Hadoop (MapReduce) • Cascading

    (tuples, workflows, job execution) • Written in Clojure • Datalog (logic programming) Thursday, May 23, 13

  10. Abstract evrthn! Thursday, May 23, 13

  11. Source where data pours from Thursday, May 23, 13

  12. Pipe that data flows through Thursday, May 23, 13

  13. Filter that makes sure that only good stuff goes through

    Thursday, May 23, 13

  14. Tuple they actually flow Thursday, May 23, 13

  15. Thursday, May 23, 13

  16. Query anatomy Thursday, May 23, 13

  17. (?<- (stdout) [?person ?person-age] (age ?person ?person-age) (< ?person-age 30))

    Thursday, May 23, 13

  18. (?<- (stdout) [?person ?person-age] (age ?person ?person-age) (< ?person-age 30))

    Output Thursday, May 23, 13

  19. (?<- (stdout) [?person ?person-age] (age ?person ?person-age) (< ?person-age 30))

    output vars Thursday, May 23, 13

  20. (?<- (stdout) [?person ?person-age] (age ?person ?person-age) (< ?person-age 30))

    Input input vars Thursday, May 23, 13

  21. (?<- (stdout) [?person ?person-age] (age ?person ?person-age) (< ?person-age 30))

    Logic/aggregations Thursday, May 23, 13

  22. Sources and sinks • HDFS (go figure) • Cassandra •

    MongoDB • SQL data sources • File system • Memory sources Thursday, May 23, 13

  23. (?<- (stdout) [?person] (age ?person 25)) Exact match of second

    element in a tuple Thursday, May 23, 13

  24. (defn younger-than? [limit age] (< age limit)) (?<- (stdout) [?person

    ?age] (age ?person ?age) (younger-than? 32 ?age)) Predicate match, fn call Predicate Thursday, May 23, 13

  25. (?<- (stdout) [?person ?count] (follows ?person _) (c/count ?count)) Aggregation

    Thursday, May 23, 13

  26. SHOWTIME! Thursday, May 23, 13

  27. Benefits •Query language is same as application language •Subqueries, reusability

    •Ad-hoc querying •Cascading underneath, so taps for all DBs work •Reuse application logic •Text editor integration Thursday, May 23, 13

  28. @ifesdjeen (twitter/github) Thursday, May 23, 13