Clojure in Big Data pipeline

Talk given in Vilnius Clojure meetup on Clojure in big data pipeline at Vinted.

Transcript

1. SPEAKER Clojure in Big Data pipeline Saulius Grigaliūnas
2. @imsaulius saulius
3. Make second-hand the first choice worldwide
4. Big Data? http://bigdatapix.tumblr.com Big data is data that exceeds the processing capacity of conventional database systems.
5. Goals • How well the product is doing? • Advanced product features using machine learning, collaborative filtering • MySQL scalability issues • Too slow for “big”-OLAP • Can’t store tracking data
6. Big data ingestion @ Vinted • up to 19 billion events / month • up to 0.7 billion events / day • growing
7. HTTP -> UDP UDP -> Kafka Kafka -> Kafka Kafka -> HDFS MySQL -> Kafka
9. A high throughput distributed messaging system •Distributed: partitions messages across multiple nodes •Reliable: messages replicated across multiple nodes •Persistent: all messages are persisted to disk •Performant: works just fine with 70 000 message writes per second. Up to 700Mb/s (when replicating/rebalancing).
10. •Broker - Kafka server •Producer - N producers send messages to Brokers •Consumer - N consumers read messages from Brokers. Each at its own pace.
11. @ Vinted • 6 Broker nodes • ~4TB of usable space • Allows us to safely keep 3+ weeks worth of event tracking data (= Hadoop can be down for 3 weeks w/ out data loss)
12. @ Vinted • 2 services in production. Deployed across 39 + 9 hosts. • 1 service in pre-production state. • 1 service in hackathon state. • 4 people proficient at writing Clojure code. • Clojure code written using Sublime, Vim, Emacs. • ~1800 LOC, ~1500 lines of test code • Still learning
13. Why Clojure? • Dynamically typed • Easier transition from Ruby • REPL++ • Ruby is an acceptable Lisp? • Eco-system • Native Avro support • Native Kafka library (JVM) • mysql-connector-java
14. Standard Clojure project @ Vinted • Small codebase (micro service?) • Continuous testing with Jenkins • Continuous deployment with Jenkins + Chef • Configuration management with Chef
15. Standard Clojure project @ Vinted • Clojure 1.6 • Leiningen • Trapperkeeper (component pattern) • Logback • clojure.test • Aiming for dev-prod parity with Vagrant + Ansible
16. Componentised services • Service consists of components • Components process stream of events in a conveyor belt principle
17. Component (defrecord Database [host port connection] ;; Implement the Lifecycle protocol component/Lifecycle (start [component] (println ";; Starting database") (let [conn (connect-to-database host port)] (assoc component :connection conn))) (stop [component] (println ";; Stopping database") (.close connection) (assoc component :connection nil)))
18. What’s next? (def prop-sorted-first-less-than-last (prop/for-all [v (gen/not-empty (gen/vector gen/int))] (let [s (sort v)] (< (first s) (last s))))) Property based testing
19. What’s next? Dynamic Clojure component configuration • Apache Zookeeper • Changing log levels • Enabling/disabling REPL at runtime
20. What’s next? Real time anomaly detection https://github.com/etsy/skyline
21. More stream processing • Build real time dashboards / reports / OLAP • Detect anomalies in event streams - identify failures quicker than we get a new support ticket • Join event streams with application metric or logging streams for root cause identification?
22. What’s next? Better & smarter product
23. Thanks! • The Log: What every software engineer should know about real- time data's unifying abstraction • All Aboard the Databus! Linkedin’s Scalable Consistent Change Data Capture Platform • Wormhole pub/sub system: Moving data through space and time • The “Big Data” Ecosystem at LinkedIn • The Unified Logging Infrastructure for Data Analytics at Twitter • Kafka: A Distributed Messaging System for Log Processing • Building LinkedIn’s Real-time Activity Data Pipeline
24. Components @ event-passage