MongoDB Transaction Pattern

Slide 1

Slide 1 text

MongoDB Transaction Pattern Who? Ian Yang From? Intridea Inc. When? 2013-05-15 Wed

Slide 2

Slide 2 text

Outline 1 Overview 2 Write Operation Properties 3 Two Phrase Commit 4 Recovering 5 Rollback 6 Multiple Applications 7 References

Slide 3

Slide 3 text

Transaction (ACID) Atomicity all or nothing. Consistency does not violate any integrity constraints Isolation allow concurrent execution of transactions Durability Transaction is persistent after commit

Slide 4

Slide 4 text

MongoDB Features Atomicity Only on the level of a single document. Write Concern Return write operation result

Slide 5

Slide 5 text

MongoDB Transaction Atomicity done or rollback Consistency rollback may leave system in inconsistent state. Isolation other transactions can see intermediate result. Durability write concern, replica set, journal.

Slide 6

Slide 6 text

Requirements of Write Operations in Transaction Revertible for recover and rollback Idempotent for resume, recover and rollback

Slide 7

Slide 7 text

Not Revertible db.users.save(user); db.users.update ({ username: "ian"}, user); db.users.update ({ username: "ian"}, {$set: {balance: 120}});

Slide 8

Slide 8 text

Revertible db.users.update ({ username: "ian"}, {$inc: {balance: 20}}); db.users.update ({ username: "ian"}, {$inc: {balance: -20}}); db.users.update ({ username: "ian"}, {$push: {tx: 20}}); db.users.update ({ username: "ian"}, {$pull: {tx: 20}});

Slide 9

Slide 9 text

Not Idempotent db.users.update ({ username: "ian"}, {$inc: {balance: 20}}); db.users.update ({ username: "ian"}, {$push: {tx: 20}}); db.users.update( {username: "ian", balance: 100}, {$inc: {balance: 20}} );

Slide 10

Slide 10 text

Idempotent db.users.update( {username: "ian", tx: {$ne: tx._id}}, {$inc: {balance: 20}, $push: {tx: tx._id}} ); db.users.update( {username: "ian", tx: tx._id}, {$inc: {balance: -20}, $pull: {tx: tx._id}} );

Slide 11

Slide 11 text

Sample Transfer 20 dollars from Ian to Daniel

Slide 12

Slide 12 text

Create Transaction Transaction provides a unique identiﬁer, enough information to recover and rollback, a document for atomic write operation.

Slide 13

Slide 13 text

var tx = { _id: ObjectId (), source: ian._id , dest: daniel._id , amount: 20 }; db.transactions.create(tx);

Slide 14

Slide 14 text

Transaction State Machine State represents transaction stages Diﬀerent actions should be taken in diﬀerent stages to recover or rollback

Slide 15

Slide 15 text

Step 1: Create Transaction var tx = { _id: ObjectId (), source: ian._id , dest: daniel._id , amount: 20, state: 'initial ' }; db.transactions.create(tx);

Slide 16

Slide 16 text

Step 2: initial → pending db.transactions.update( {_id: tx._id , state: 'initial '}, {$set: {state: 'pending '}} );

Slide 17

Slide 17 text

Step 3: Apply Transaction db.users.update( {_id: tx.source , tx: {$ne: tx._id}}, {$push: {tx: tx._id}, $inc: { balance: - tx.amount }} ); db.users.update( {_id: tx.dest , tx: {$ne: tx._id}}, {$push: {tx: tx._id}, $inc: { balance: tx.amount }} );

Slide 18

Slide 18 text

Step 4: pending → committed db.transactions.update( {_id: tx._id , state: 'pending '}, {$set: {state: 'committed '}} );

Slide 19

Slide 19 text

Step 5: Remove Pending Transaction db.users.update( {_id: tx.source , tx: tx._id}, {$pull: {tx: tx._id}} ); db.users.update( {_id: tx.dest , tx: tx._id}, {$pull: {tx: tx._id}} );

Slide 20

Slide 20 text

Step 6: committed → done db.transactions.update( {_id: tx._id , state: 'committed '}, {$set: {state: 'done'}} );

Slide 21

Slide 21 text

Recover Initial Transactions Recover from Step 2: initial → pending

Slide 22

Slide 22 text

Recover Pending Transactions Recover from Step 3: Apply Transactions

Slide 23

Slide 23 text

Recover Committed Transactions Recover from Step 5: Remove Pending Transaction

Slide 24

Slide 24 text

General Rule Recover from the Step just following the step transaction just entered current state. State transition creates a check point

Slide 25

Slide 25 text

Rollback Operations Better to create a new transaction in reverse direction for Committed Transactions and Done Transactions Use following steps to rollback initial and pending transactions

Slide 26

Slide 26 text

Rollback 1: state → canceling db.transactions.update( {_id: t._id , state: {$in: ['initial ', ' pending ']}}, {$set: {state: 'canceling '}} );

Slide 27

Slide 27 text

Rollback 2: Undo the Transaction db.users.update( {_id: tx.dest , tx: {$ne: tx._id}}, {$pull: {tx: tx._id}, $inc: { balance: - tx.amount }} ); db.users.update( {_id: tx.source , tx: tx._id}, {$pull: {tx: tx._id}, $inc: { balance: tx.amount }} );

Slide 28

Slide 28 text

Rollback 3: canceling → canceled db.transactions.update( {_id: t._id , state: {$in: ['canceling ']}}, {$set: {state: 'canceled '}} );

Slide 29

Slide 29 text

Recover Rollback Restart from Rollback 2 for canceling transactions

Slide 30

Slide 30 text

Conclusion Create a document for any transaction Store transaction info in the document Apply only Revertible and Idempotent operations

Slide 31

Slide 31 text

Conclusion (Cont.) Transit state to save current work Transit state if several operation branches exists

Slide 32

Slide 32 text

Enough? Only 1 application can handle a give transaction at any point in time.

Slide 33

Slide 33 text

Background A picks a pending transaction to recover B picks also a pending transaction to recover

Slide 34

Slide 34 text

Scenario 1 A is so slow, when it starts apply transaction, B already has done the transaction. A must rollback. If A exists abnormally, the transaction is in inconsistent state.

Slide 35

Slide 35 text

Scenario 2 B just ﬁnished Step 5: Removing Pending Transaction A applied transactions and exited abnormally B change state to done without error

Slide 36

Slide 36 text

Solution findAndModify lock db.transactions.findAndModify ({ query: {state: "initial", application: {$exists: 0}}, update: {$set: {state: "pending", application: "A1"}}, new: true });

Slide 37

Slide 37 text

Lock Release db.transactions.findAndModify ({ query: {application: "A1"}, update: {$unset: {application: 1}}, new: true });

Slide 38

Slide 38 text

Lock Forever? Application exists abnormally before release the lock Solution: timeout

Slide 39

Slide 39 text

Timeout Lock now = new Date(); timeout = new Date(now.valueOf () + 60 * 10 * 1000); db.transactions.findAndModify ({ query: {state: "initial", lockUntil: {$not: {$gte: now}}}, update: {$set: {state: "pending", lockUntil: timeout}}, new: true });

Slide 40

Slide 40 text

Client Requirements Client timeout to ensure the operation is ﬁnished in expected time. Check time after return from block IO Defer transactions cleanup

Slide 41

Slide 41 text

References http://docs.mongodb.org/ manual/tutorial/ perform-two-phase-commits/ http://docs.mongodb.org/ manual/tutorial/ isolate-sequence-of-operations/

Slide 42

Slide 42 text

Q & A