Multi-Master Replication with MongoDB Rick Copeland @rick446 Tuesday, December 4, 12 

The Client Problem MongoDB (factory floor) MongoDB (cloud) App Server App Server App Server App Server Camera Camera Camera Camera Frame & Feature Data (mostly) Inspection Metadata (mostly) • Cameras save frame images to GridFS • Inspections, measurements, etc. defined by metadata in MongoDB • Access locally + from the web Tuesday, December 4, 12 

The Client Problem MongoDB (factory floor) MongoDB (cloud) App Server App Server App Server App Server Camera Camera Camera Camera Frame & Feature Data (mostly) Inspection Metadata (mostly) • Frames must still be saved, inspections performed • Tolerate being inaccessible from the cloud Tuesday, December 4, 12 

Put the Primary in the Cloud? MongoDB (factory floor) MongoDB (cloud) App Server App Server App Server App Server Camera Camera Camera Camera Frame & Feature Data (mostly) Inspection Metadata (mostly) Primary • Pros: • More conventional • Additional seconaries in the cloud • Cons: • Dependent on internet • Saving frame data up an ADSL connection??!! Tuesday, December 4, 12 

Put the Primary in the Factory? MongoDB (factory floor) MongoDB (cloud) App Server App Server App Server App Server Camera Camera Camera Camera Frame & Feature Data (mostly) Inspection Metadata (mostly) Primary • Pros: • Frame data saves fast • Resilient to outages • Cons: • Backups / durability? • Seems wrong, somehow... Tuesday, December 4, 12 

But what if.... MongoDB (factory floor) MongoDB (cloud) App Server App Server App Server App Server Camera Camera Camera Camera Frame & Feature Data (mostly) Inspection Metadata (mostly) Primary Primary • Pros: • Frame data saves fast • Resilient to outages • Backups / durability in cloud • Cons: • But MongoDB doesn’t do multi-master replication Tuesday, December 4, 12 

Subverting Replication Internals: Bending the Oplog to your Will Tuesday, December 4, 12 

Replication: the Oplog { "ts" : Timestamp(1317653790000, 2), "h" : -6022751846629753359, "op" : "i", "ns" : "confoo.People", "o" : { "_id" : ObjectId("4e89cd1e0364241932324269"), "first" : "Rick", "last" : "Copeland” } } Tuesday, December 4, 12 

Replication: the Oplog { "ts" : Timestamp(1317653790000, 2), "h" : -6022751846629753359, "op" : "i", "ns" : "confoo.People", "o" : { "_id" : ObjectId("4e89cd1e0364241932324269"), "first" : "Rick", "last" : "Copeland” } } Insert Tuesday, December 4, 12 

Replication: the Oplog { "ts" : Timestamp(1317653790000, 2), "h" : -6022751846629753359, "op" : "i", "ns" : "confoo.People", "o" : { "_id" : ObjectId("4e89cd1e0364241932324269"), "first" : "Rick", "last" : "Copeland” } } Insert Collection name Tuesday, December 4, 12 

Replication: the Oplog { "ts" : Timestamp(1317653790000, 2), "h" : -6022751846629753359, "op" : "i", "ns" : "confoo.People", "o" : { "_id" : ObjectId("4e89cd1e0364241932324269"), "first" : "Rick", "last" : "Copeland” } } Insert Collection name Object to insert Tuesday, December 4, 12 

Step 1: Build Triggers def run(self): while True: yield self.checkpoint spec = dict(ts={'$gt': self.checkpoint}) q = self._oplog.find( spec, tailable=True, await_data=True) found=False for op in q.sort('$natural'): found = True self.checkpoint = op['ts'] for callback in self._callbacks.get( (op['ns'], op['op']), []): callback(**op) if found: sleep(0) else: sleep(1) Tuesday, December 4, 12 

Step 1: Build Triggers def run(self): while True: yield self.checkpoint spec = dict(ts={'$gt': self.checkpoint}) q = self._oplog.find( spec, tailable=True, await_data=True) found=False for op in q.sort('$natural'): found = True self.checkpoint = op['ts'] for callback in self._callbacks.get( (op['ns'], op['op']), []): callback(**op) if found: sleep(0) else: sleep(1) Find ops Tuesday, December 4, 12 

Step 1: Build Triggers def run(self): while True: yield self.checkpoint spec = dict(ts={'$gt': self.checkpoint}) q = self._oplog.find( spec, tailable=True, await_data=True) found=False for op in q.sort('$natural'): found = True self.checkpoint = op['ts'] for callback in self._callbacks.get( (op['ns'], op['op']), []): callback(**op) if found: sleep(0) else: sleep(1) Find ops Call triggers Tuesday, December 4, 12 

Step 2: Build the replication MMM_REPL_FLAG = '__mmm' def trigger(ts, h, op, ns, o, o2=None, b=False): if op == 'i': if o.get(MMM_REPL_FLAG) == self.id: return o.setdefault(MMM_REPL_FLAG, src_id) collection.insert(o) elif op == 'u': upsert = b if any(k.startswith('$') for k in o): # With modifiers, check & update setters setters = o.setdefault('$set', {}) else: # Without modifiers, update doc directly setters = o if setters.get(MMM_REPL_FLAG) == self.id: return setters.setdefault(MMM_REPL_FLAG, src_id) collection.update(o2, o, upsert) elif op == 'd': justOne = b collection.remove(o) Tuesday, December 4, 12 

Step 2: Build the replication MMM_REPL_FLAG = '__mmm' def trigger(ts, h, op, ns, o, o2=None, b=False): if op == 'i': if o.get(MMM_REPL_FLAG) == self.id: return o.setdefault(MMM_REPL_FLAG, src_id) collection.insert(o) elif op == 'u': upsert = b if any(k.startswith('$') for k in o): # With modifiers, check & update setters setters = o.setdefault('$set', {}) else: # Without modifiers, update doc directly setters = o if setters.get(MMM_REPL_FLAG) == self.id: return setters.setdefault(MMM_REPL_FLAG, src_id) collection.update(o2, o, upsert) elif op == 'd': justOne = b collection.remove(o) Inserts Tuesday, December 4, 12 

Step 2: Build the replication MMM_REPL_FLAG = '__mmm' def trigger(ts, h, op, ns, o, o2=None, b=False): if op == 'i': if o.get(MMM_REPL_FLAG) == self.id: return o.setdefault(MMM_REPL_FLAG, src_id) collection.insert(o) elif op == 'u': upsert = b if any(k.startswith('$') for k in o): # With modifiers, check & update setters setters = o.setdefault('$set', {}) else: # Without modifiers, update doc directly setters = o if setters.get(MMM_REPL_FLAG) == self.id: return setters.setdefault(MMM_REPL_FLAG, src_id) collection.update(o2, o, upsert) elif op == 'd': justOne = b collection.remove(o) Inserts Updates Tuesday, December 4, 12 

Step 2: Build the replication MMM_REPL_FLAG = '__mmm' def trigger(ts, h, op, ns, o, o2=None, b=False): if op == 'i': if o.get(MMM_REPL_FLAG) == self.id: return o.setdefault(MMM_REPL_FLAG, src_id) collection.insert(o) elif op == 'u': upsert = b if any(k.startswith('$') for k in o): # With modifiers, check & update setters setters = o.setdefault('$set', {}) else: # Without modifiers, update doc directly setters = o if setters.get(MMM_REPL_FLAG) == self.id: return setters.setdefault(MMM_REPL_FLAG, src_id) collection.update(o2, o, upsert) elif op == 'd': justOne = b collection.remove(o) Inserts Updates Deletes Tuesday, December 4, 12 

Step 3 Tuesday, December 4, 12 

But be careful if you use it.... Tuesday, December 4, 12 

Caveats • Conflicts? • Performance? • Reliability? • Support? Tuesday, December 4, 12 

github.com/rick446/mmm Tuesday, December 4, 12 

Client Results Tuesday, December 4, 12 

Questions? MongoDB MultiMaster: http://github.com/rick446/mmm Rick Copeland @rick446 http://arborian.com Tuesday, December 4, 12