Event Logging at Belly

Using Fluentd to move logs into HDFS, S3, and RabbitMQ

Transcript

1. Kevin Reedy @kevinreedy kevin@bellycard.com http://tech.bellycard.com Event Logging at Belly ! Using fluentd to move logs into HDFS, S3, and RabbitMQ
2. Event Logging at Belly @kevinreedy What is Belly?
3. Event Logging at Belly @kevinreedy
4. The Nation’s Largest Loyalty Network “From Sea To Shining Sea” - literally. Tap into Belly’s nationwide network of millions of active Members and thousands of Merchant Partners. Interact with more customers better. Tens of Millions Of Visits Millions Of Members Thousands Of Locations
5. Event Logging at Belly @kevinreedy What is Belly? •A digital customer loyalty platform •One card or app for the entire network •Over 2,000,000 members •Over 8,000 merchants •Over 15,000,000 visits •Unique rewards
6. Event Logging at Belly @kevinreedy Who is this Kevin guy • Grew up “in Chicago” • Networks and Telecom degree from DePaul University • Infrastructure Architect on the very small systems team at Belly • At Belly for a year • Previously at Tap.Me, OpenDNS, Riverbed
7. Event Logging at Belly @kevinreedy Event Logging At Belly
8. Event Logging at Belly @kevinreedy Event Logging - Inputs •User Events •Client Events •API Events •Business Events
9. Event Logging at Belly @kevinreedy User Events •Mobile Applications •In-Store Tablet Application •Homepage
10. Event Logging at Belly @kevinreedy Client Events •In-store tablet •Heartbeats •Error Logs
11. Event Logging at Belly @kevinreedy API Events • User Creation • Checkins • Reward Redemption • Email • Send • Delivered • Opened • Clicked • Spam Reported
12. Event Logging at Belly @kevinreedy Business Events •Salesforce Callbacks •Meetings •Installs •Closes
13. Event Logging at Belly @kevinreedy Event Logging - Outputs •Hadoop HDFS •S3 •RabbitMQ
14. Event Logging at Belly @kevinreedy FluentD •Open-source tool to collect events and logs •Code and Plugins written in Ruby •Everything is JSON •150+ plugins, including Hadoop, S3, MongoDB, Elasticsearch •Provides Disk and RAM Buffering
15. Event Logging at Belly @kevinreedy Bee •Data Validation and Sanitization Service •Written in Ruby •Rules are defined in YAML •Events from untrusted sources must first pass through Bee •This includes mobile clients
16. Event Logging at Belly @kevinreedy Bee
17. Event Logging at Belly @kevinreedy Putting it all together
18. Event Logging at Belly @kevinreedy API Layer •API Servers and Bee Servers run Fluentd locally •Fast response time •Buffering locally if there are issues •Fluentd also has a UDP input if you’d rather have best effort delivery
19. Event Logging at Belly @kevinreedy API ➡ Aggregation Layer • Each API Fluentd instance forwards to two sets of servers: • Fluentd aggregators in Chicago • Fluentd aggregators in EC2 us-east-1 • Currently each aggregator set is 2 hosts • API Fluentd instances treat each aggregation set as Master/Master, but this is configurable
20. Event Logging at Belly @kevinreedy Aggregation Layer •Limits the number of connections to Output Layer •Provides disk buffering for S3 and HDFS Outputs •Provides minimal buffering for RabbitMQ Outputs
21. Event Logging at Belly @kevinreedy Aggregation ➡ Output Layer • Forest Output Plugin • Output Plugins for Fluentd • S3 • https://github.com/fluent/fluent-plugin-s3 • HDFS • https://github.com/fluent/fluent-plugin-webhdfs • HttpsFs instead of WebHDFS • RabbitMQ • https://github.com/restorando/fluent-plugin-amqp
22. Event Logging at Belly @kevinreedy Applications •Ad-hoc queries in Hive •Hiverunner •User Facts Service •Honey Reports •Checkins Map
23. Event Logging at Belly @kevinreedy Honey Reports
24. Event Logging at Belly @kevinreedy Honey Reports
25. Event Logging at Belly @kevinreedy Checkins Map
26. Event Logging at Belly @kevinreedy Future Plans •Evaluate using RabbitMQ as the Aggregation Layer •Application Logs as an Input •Splunk or Elasticsearch / Kibana as Outputs
27. Event Logging at Belly @kevinreedy Questions? ! ! …if there is time
28. Event Logging at Belly @kevinreedy Keep in touch •twitter: @kevinreedy •email: kevin@bellycard.com •internets: http://tech.bellycard.com