TokuMX for Analytics

Presented at TokuMX Seattle Meetup
This talk details on the TokuMX deployment at Brightcove (Analytics) and the lessons we learnt.

Transcript

1. MongoDB -> TokuMX An analytics prespective
2. Who am I? • Software engineer @ Brightcove Inc. ◦ With reporting & analytics team. • Work on distributed systems • Near* real-time and explorative analytics systems • I also love ops (more like eliminating ops)
3. What is • TLDR; An online video platform • Mostly targeted at professional content ◦ Paying customers (not UGC*) ▪ Interested in monetizing ▪ Analyze video trends ▪ Predict “what works”
4. What is RNA all about • Real Time* analytics • Billing reports and Invoicing. • BI • Trend analysis. • Experiments and hypothesis* validations. • … whatever you want to do with data streams.
5. Architecture V I D E O P L A Y E R S Real time data set API UI BI Billing Historical Data (Append only) Batch Inserter Long term log storage - AWS S3 Log processing Joins / data cleanup Batch Processing layer (MR) Collectors Updated content / Metrics (Fancy stuff) External Data sources
6. Constraints • Tight Real time SLA ◦ 1 minute is the limit - Usually around 5 - 10 seconds ▪ 2 mins is status blue ▪ > 6 mins is yellow • High cardinality data • High volume • Budget constraints • Drill down abilities
7. SPIKE #1: Explore them databases • Other contenders ◦ MySQL/RDBMS ◦ MongoDB ◦ Roll our own … ◦ Druid
8. MongoDB Wins • Awesomely fast writes • Awesomely fast reads • Not too much ops overhead • The birds were shining the sun was chirping or was it ….
9. Fast forward 6 months Problems start • Repl log is not free • Chunk migrations will kill you • A server restart may kill your oplog • Large datasets will kill you • Indexes will kill you (again) • Even deletes will kill you!
10. MongoDB is a bad (girl/boy)friend • You are still in love with em’ but don’t know why! ◦ Simple to understand ◦ Query language makes sense ◦ Freedom to evolve schemas • But we were spending most of our time doing OPS.
11. • I love reading papers! • Research: What went wrong? ◦ Indexes too big ◦ Replication is janky! ◦ We live in the real-world! ◦ Mongo support in general was not very helpful. SPIKE # 2: Find a database that works!
12. Options • Stay with mongoDB • RethinkDB • Cassandra • MySQL/RDBMS • Make our own • Riak • …..
13. The story goes 1. Michael A. Bender, Martín Farach-Colton, Jeremy T. Fineman, Yonatan Fogel, Bradley C. Kuszmaul, and Jelani Nelson. "Cache-Oblivious Streaming B-Trees." Proceedings of the 19th ACM Symposium on Parallelism in Algorithms and Architectures (SPAA), pages 81-92, 2007. [PDF] [postscript] 2. Fractal Trees - Wikipedia 3. TokuDB 4. Many talks, github source code reading …. • Tokutek (specifically @leifwalsh, @zkasheff, @tmcallaghan) seemed to know what they were doing. • The benchmarks looked good. Proceed to stage 2 …
14. Convincing the team * “Benchmarks lie” - <insert dev name here> * Numbers don’t lie …
15. But the big kicker! Compression! compression! compression!
16. Still not convinced... • The support was really awesome ◦ Real developers reply and super fast! ◦ “These guys know what they are doing!” ▪ We “cancelled” our Mongo service contract. • More bang for your buck ◦ Disk space - SSD are still expensive ◦ Better RAM utilization ◦ Transactions! (un-sharded only) ◦ Backups like a boss!
17. Back to analytics • Worked closely to optimize our data model ◦ Big insert batches make up for faster repl log ◦ Index behaves differently ◦ More CPU is needed for compression • We don’t add replicas like we used to ◦ HotBackup -> fastsync -> done! • We don’t add batches as we used to ◦ BulkLoad command
18. Query side! • Much lesser I/O = better perf • Clustered index = index has your data too • Fractal tree + compression = adding a new index doesn’t mean x 2 storage
19. Lessons learned • Sharding helps with writes • Replicas help with reads • Data model makes a huge difference • If you have lots of data and not enough RAM, use FS page cache (compressed) and set --cacheSize • Get more CPU - that’s a good thing! • Partitioned collections help deletes. • There are more …
20. Missing stuff • Bulk load works only in unsharded collections • Initial sync of replicas is still uncompressed • Replication algorithm is still not correct ◦ Ark algorithm impl. is in a branch • Transactions via the driver needs squinting.