Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Lessons from Building Distributed RocksDB

Lessons from Building Distributed RocksDB

Presented as part of Geeknight Chennai - November 2016

Video - https://www.youtube.com/watch?v=PSCa9_Avne0
References - https://github.com/ashwanthkumar/distributed-rocksdb-talk

D90acaa01cb59a2b8b7e986958953eee?s=128

Ashwanth Kumar

November 24, 2016
Tweet

Transcript

  1. ashwanth kumar @_ashwanthkumar Building Distributed RocksDB

  2. overview - What’s RocksDB - Why Distributed RocksDB - Lessons

    from running it in production
  3. rocksdb

  4. From facebook Fast persistent KV store Server Workloads Embeddable Optimized

    for SSDs rocksdb Fork of LevelDB Modelled after BigTable LSM Tree based SST files Written in C++
  5. Simple C++ API Has bindings in C Java Go Python

    rocksdb
  6. but why build a distributed database?

  7. data system paradigms

  8. sum of numbers

  9. sum of numbers select col from table;

  10. sum of numbers select col from table; <<results>>

  11. sum of numbers select col from table; <<results>>

  12. sum of numbers (attempt 2)

  13. sum of numbers select sum(col) from table;

  14. sum of numbers select sum(col) from table;

  15. sum of numbers <<result>> select sum(col) from table;

  16. data shipping function shipping select sum(col) from table; select col

    from table;
  17. but why ship functions?

  18. data locality for low latency / data intensive applications

  19. recursive reduction aggregations in distributed systems

  20. recursive reduction select sum(col) from table;

  21. recursive reduction select sum(col) from table;

  22. recursive reduction select sum(col) from table;

  23. recursive reduction select sum(col) from table; <<result>>

  24. - sum / multiplication - (sorted) top-K elements - operations

    on a graph - eg. link reach on twitter graph - function should be associative and optionally commutative recursive reduction
  25. rocksdb @indix

  26. - Serving our API in production for 2+ years -

    Search on hierarchical documents - Dynamic fields didn’t scale well on Solr - Brand / Store / Category Counts for a filter - Price History Service - More than a billion prices and serve online to REST queries rocksdb @indix
  27. - Stats (as Monoids) Storage System - All we want

    was approximate aggregates real-time - HTML Archive System - Stores ~120TB of url and timestamp indexed HTML pages - Real-time scheduler for our crawlers - Finds out which of the 20 urls to crawl now out of 3+ billion urls - Helps crawler crawl 20+ million urls everyday rocksdb @indix
  28. suuchi toolkit for building distributed function shipping applications github.com/ashwanthkumar/suuchi

  29. ops lessons running rocksdb in production

  30. leveled compaction

  31. https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide Tuning RocksDB for Read amplification Write amplification Space amplification

    compaction + other tuning
  32. backup & restore - Incremental backups - Store backups in

    S3 - Sometimes high CPU during backups - Restore happens outside the app lifecycle https://github.com/indix/rocks
  33. questions? https://github.com/ashwanthkumar/distributed-rocksdb-talk