Building a Data Platform with Ruby glue

Transcript

1. Building a data platform with Ruby glue Augusto Becciu
2. Who am I? • @abecciu {github, twitter} • Startups guy • Like to write and run code • Cur: CTO at • Prev: Dev / Ops Engineer at
3. Data is the DNA of a Company
4. What is a data platform?
5. Data Applications Fraud detection Monitoring and alerting Metric dashboards Recommendation systems Bidding systems Audit trails Marketing automation A/B Testing User behavior analytics
7. Two groups of applications • Real-time processing (ex. fraud detection, bidding systems, monitoring and alerting, etc. ). • Batch processing (ex. recommendations systems, user behavior analytics, etc.).
9. Design Principles • Make it super easy for developers to produce and consume data. • Optimize for low operations and infrastructure costs. • Make the best effort to not loose data, but don’t worry about guarantees. • Be programming language agnostic.
10. Data Acquisition
11. Data Standardization The unit of data is an event represented as a JSON document.
12. Data Standardization Anatomy of an event:
13. Data Consolidation
14. • Open source tool to collect events and logs. • Pluggable inputs and outputs. • Code and plugins written in Ruby. • Provides fault tolerance and simple HA facilities. • JSON everywhere.
17. Data Storage • Data stream is partitioned by day and stored in a hierarchical way like: ! • Events are saved in newline-delimited text files. • Every file is gzipped. • Events are buffered in local filesystem and uploaded to S3 periodically. /year/month/day/events.gz
18. Data Distribution • Publish-subscribe pattern. • Events are published to a topic exchange. • Consumers can subscribe to topics based on event keys. Ex. ui.webapp.# • SSL-based authentication and encryption over persistent TCP connections allow consumers to be anywhere on the Internet.
19. Tools to produce data • Libraries were created to abstract the process of creating and publishing an event. • Multiple languages supported like: Ruby, Javascript and Objective-C. For example, in a Rails app it’s as simple as: (Namespace and context data are added automagically...)
20. Tools to consume data • Any application can use any standard AMQP or S3 client • We also created a couple of command-line tools for easy prototyping and experimentation.
21. Tools to consume data tail_f gets you events from the firehose $ tail_f ui.webapp.#, errors.webapp.# | ruby do_staff.rb do_staff.rb:
22. Tools to consume data s3_cat downloads cold data from S3 $ s3_cat 2013-11-26..2013-11-28 | ruby do_staff.rb $ s3_cat 2013-10, 2013-11-01..2013-11-15 > data $ s3_cat 2013 | mongoimport -d mydb -c mycollection
23. Resources • http://fluentd.org/ • https://github.com/fluent/fluent-plugin-s3 • https://github.com/restorando/fluent-plugin-amqp • http://www.rabbitmq.com/ • https://github.com/restorando
24. Thanks! / Questions? Meet the team http://restorando.com/joinus