Distributed locking with MongoDB

During the talk problems of synchronization in the cluster are discussed. Why `synchronized` doesn't work (and why it does)? How to get distributed locking without ZooKeeper? The talk explores implementation options of synchronization primitives based on MongoDB.

Keynote slides (with gifs!): https://www.dropbox.com/s/jw7gmo1fs81oezw/distributed-locking-with-mongodb.key?dl=1

Transcript

2. Distributed locking with MongoDB Andrew Barchuk
3. Concurrency is hard…
4. …concurrency is unavoidable ̼performance ̼multi-user system ̼multi-machine environment ̼flow control ̼GUI 4
5. Things get even harder in cluster ̼more concurrency! ̼harder coordination 5
6. Why doesn’t synchronized work?
8. When synchronized does work ̼probabilistic approach ̼in-request synchronisation 8
9. 6 RPS 9 1 RPS 1 RPS 1 RPS 1 RPS 1 RPS 1 RPS
12. MongoDB concurrency control ̼unique indices ̼update-if-current (compare-and-swap) 12
13. Unique index db.users.createIndex({email: 1}, {unique: true}) MongoDB concurrency control 13
14. Unique index WriteResult({ "nInserted" : 0, "writeError" : { "code" : 11000, "errmsg" : "E11000 duplicate key error index: database.users.$email_1 dup key: { : \”user@example.com\" }" } }) 14 MongoDB concurrency control
15. Update-if-current var user = db.users.findOne({userId: 42}); var oldKarma = user.karma; var result = db.users.update( { userId: 42, karma: oldKarma }, { $inc: {karma: -1} } ); if( result.nMatched === 0 ) { // query new user and retry } 15 MongoDB concurrency control
16. Unique index: even greater …prevents duplicate insertions or updates that result in duplicate values: update() and findAndModify() 16 MongoDB concurrency control
18. Here we go: MongoDB based distributed lock
19. acquire()
20. acquire() release() Critical section }
21. Acquiring lock var result = db.locks.update( { id: 42 }, {}, { upsert: true } ); if (result.nUpserted === 1) { // Yay! Lock acquired! } else { // retry… } Lock implementation 21
22. Preparing index db.locks.createIndex({id: 1}, {unique: true}) Lock implementation 22
23. Releasing lock db.locks.remove({id: 42 }) 23 Lock implementation
24. That’s all!
25. …actually, no
27. What if a lock is not released?
28. Cleaning up db.locks.update( { id: 42 }, { aquiredAt: new Date() }, { upsert: true } ) Lock implementation 28
29. TTL Index db.locks.createIndex({acquiredAt: 1}, {expireAfterSeconds: 30}) Lock implementation 29
30. How to put a thread to sleep? …and how to wake it up?
31. Goals ̼do not waste resources (network and CPU time) ̼minimize latency Putting the thread to sleep and waking it up 31
32. Putting the thread to sleep naive solution: do not put the thread to sleep 32
33. Putting the thread to sleep “smart” solution: tune Thread.sleep() intervals 33
34. Putting the thread to sleep intelligent solution: capped collection + tailableCursor + awaitData 34
35. db.createCollection(“locks”, {capped: true, size: 1000}) 35 Improved lock implementation Capped collection
36. db.locks.find( { id: 42, acquired: false } ).addOption(DBQuery.Option.tailable + DBQuery.Option.awaitData) 36 Improved lock implementation Blocking the thread
37. Trying to acquire the lock var result = db.locks.update( { id: 42, acquired: false }, { $set: {acquired: true} } ); if (result.nUpserted === 1 || result.nModified === 1) { // lock acquired } Improved lock implementation 37
38. Releasing the lock db.locks.update( { id: 42 }, { $set: {acquired: false} } ) Improved lock implementation 38
39. Putting it all together while(!tryAcuire()) { waitForRelease(); } // do your stuff, lock acquired release(); Improved lock implementation 39
40. Not perfect though ̼Not compatible with TTL indexes ̼Tailable cursor dies after some time without data ̼Capped collection size is fixed Improved lock implementation 40
41. Let’s have fun! And implement a semaphore Coz we can!
42. Acquiring semaphore var result = db.locks.update( { id: 42, count: {$gt: 0} }, { $setOnInsert: {count: 5}, $inc: {count: -1} }, { upsert: true } ); if (result.nUpserted === 1 || result.nModified === 1) { // semaphore acquired } 42 Semaphore implementation
43. Acquiring semaphore var result = db.locks.update( { id: 42, count: {$gt: 0} }, { // $setOnInsert: {count: 5}, until SERVER-10711 fix $inc: {count: -1} }, { upsert: true } ); if (result.nUpserted === 1 || result.nModified === 1) { // semaphore acquired } 43 Semaphore implementation
44. Releasing semaphore var result = db.locks.update( { id: 42 }, { $inc: {count: 1} } ); 44 Semaphore implementation
45. Considerations ̼use basic concurrency control whenever possible ̼MongoDB may be not perfect for your use case ̼always review concurrency related code (better twice!) 45
46. Resources ̼https://goo.gl/AIn9hd ̼https://speakerdeck.com/raindev/distributed-locking-with-mongodb ̼http://docs.mongodb.org/manual/faq/concurrency/ ̼http://docs.mongodb.org/manual/core/write-operations-atomicity/ 46
47. Thank you everybody!
48. Questions are welcomed!
49. 49 Andrew Barchuk Software Engineer at Yandex Media Services Contacts @raindev_ @raindev andrew@raindev.io