Erich Nachbar

Eric Nachbar, CTO of Quanitfind gives a presentation on Pub/Sub and Kafka. Part of the Code Social Presentation Series. Watch the video for this presentation here: http://mrkn.co/qifbi

Transcript

1. Scale with Pub/Sub Erich Nachbar
2. Overview • About Quantifind • Pub/Sub Goodness • Apache Kafka • Q & A
3. About Quantifind • Combines signals from unstructured data (Twitter, Amazon, Facebook,...) with Structured data (company financials, product sales, ratings, ...) • Automatically extracts drivers to predict future behavior
4. Why Pub/Sub? Example: Tracking User Actions • Show Real-time monitoring dashboard • Save them to Hadoop for analysis • Replay Actions for Load Tests • Take Hadoop offline for maintenance Producers Consumers Point-to-Point
5. Publish / Subscribe Producers Data 1 Data 2 Consumers Data 1,2 Data 1,2 Topic Usage: Data Distribution Loose Coupling Increased Robustness
6. Queues Queue Producers Data 1 Data 2 Data 1 Data 2 Consumers Usage: Work Distribution Increased Robustness
7. Apache Kafka Many Desirable Properties • Supports Pub/Sub & Queues (aka Groups) • Super simple, cross-language client API • Very high throughput of messages • Scales by adding brokers (Zookeeper) • Keeping large message backlog is cheap and does not cause slowdowns • Battle-tested in production at LinkedIn
8. Summary • Look for Pub/Sub patterns early avoiding pain later • Every Pub/Sub system behaves different Make sure to try them under your extreme use cases • Choose your Pub/Sub provider wisely If your Pub/Sub is down... • Kafka is awesome
9. Want to help? • Looking for frontend & backend engineers • Stack HTML5/CSS3/jQuery/Grails Scala/Storm/Spark/Cassandra/Kafka (no, we don’t expect you to know them all) • Funded by Andreessen Horowitz & Redpoint
10. Resources Apache Kafka bit.ly/kafkaPubSub Kafka Clients bit.ly/kafkaClients
11. Q & A