Scaling MongoDB to 10,000 rps & beyond

We use MongoDB heavily as our primary database. Here we talk about the operational issues and challenges we faced for scaling MongoDB to 10,000 rps & beyond

Transcript

1. Scaling MongoDB to 10,000 requests/sec Abhishek Amberkar DevOps at Helpshift @greenmang0
2. Or, Running MongoDB in Production Lessons learnt @ Helpshift
3. What’s MongoDB MongoDB is an open-source document database that provides high performance, high availability, and automatic scaling. A MongoDB document
4. Why MongoDB • Schemaless • Faster Development in Early Stage • Easier to get started with compared to other NoSQL Solutions
5. Master Slave Replication • Easier to maintain • No automatic failover • Compulsory downtime for all major administrative operations ◦ convert master-slave to replica set ◦ promote slave to be a master ◦ inverting master and slave
6. Global Locks • 2.2 - Global locks for all write operations • 2.4 - “database” level locking • Global locks are only for certain operations • No “row” level locking yet
7. Replica Sets • Automatic failover • Writes were lost without Write Concern • Use replica acknowledge • Priorities • Promotions and step downs
8. Sharding • Scale writes efficiently • Choose right “shard key” to avoid “hot regions” • Different ways of taking backups in sharded environment
9. Read Preference • Read operations return results from the primary • rs.slaveOk() / secondaryPreferred • Propagation delays • Stale data • Eventual Consistency • Write Concern
10. Read Preference…. • Maintaining an availability during a failover • Tags can be added to secondaries ◦ “tags” : { “dc” : “east”, “disk”: “ssd” } • Tags can be used with “write concern” as well
11. Monitoring • MMS by mongodb.org • db.serverStatus()
13. Backups • mongodump • Paid backup service by mongodb.org
14. Miscellaneous • Working set needs to fit in memory "workingSet": { "pagesInMemory": 315253 } • repairDatabase() • XFS • AWS EC2 - LVM + RAID10 • sysctl.conf ◦ net.ipv4.tcp_keepalive_time = 300
15. Future Directions (?) • TokuMX ◦ Transactions! ◦ 90% space reduction ◦ Drop in replacement for MongoDB 2.4 ◦ 10x Faster Indexing ◦ Reduced SSD wear ◦ No “Text Search” ◦ No “Geospatial Indexes”
16. Questions?
17. We’re hiring! jobs@helpshift.com