MongoNYC 2012: Building a MongoDB Power Chat Server

Embed

Start on current slide

Slide 1

Slide 1 text

Building an IRC service with MongoDB irc1.10gen.cc / 6667 / #fun Github: erh/mongo-irc Eliot Horowitz, CTO Jared Rosoff, Product Marketing Edouard Servan-Schreiber, Solution Architecture

Slide 2

Slide 2 text

• 10:00 – 10:40 App Design, Schema Design, MessageBus on MongoDB • 10:45 – 11:25 Deployment, Replication, Monitoring • 11:40 – 12:20 Scaling Message Logging and Sharding – Live! • 12:25 – 1:05 Backups, Monitoring and Operations for a Sharded Cluster Agenda irc1.10gen.cc / 6667 / #fun

Slide 3

Slide 3 text

In this talk • We’re building an IRC server • Architectural overview • Data model overview • Designing a Message Bus • Demo the app irc1.10gen.cc / 6667 / #fun

Slide 4

Slide 4 text

What is IRC? irc1.10gen.cc / 6667 / #fun

Slide 5

Slide 5 text

Data Model • Rooms • Users • Messages • Servers irc1.10gen.cc / 6667 / #fun

Slide 6

Slide 6 text

Schema Design • Rooms { "_id" : "#fun", "topic" : "not set", "created" : ISODate("2012-05-21T19:53:42.690Z") } • Servers { "_id" : "ip-10-190-215-182:6667", "last_ping" : ISODate("2012-05-21T20:15:55.379Z") } irc1.10gen.cc / 6667 / #fun

Slide 7

Slide 7 text

MessageBus { "_id" : ISODate("2012-05-21T19:53:42.691Z"), "type" : "room", "room" : "#fun", "message" : ":edouardss!~edouardss@0 JOIN #fun", "incarnation" : "Mon May 21 19:52:03 UTC 2012-0.7612059410121808" } { "_id" : ISODate("2012-05-22T18:29:15.950Z"), "type" : "room", "room" : "#fun", "message" : ":edouardss!~edouardss@0 PRIVMSG #fun :hello mongonyc", "incarnation" : "Tue May 22 18:19:25 UTC 2012-0.5635396421023062" } irc1.10gen.cc / 6667 / #fun

Slide 8

Slide 8 text

Data for Single Server Setup • Keep track of rooms • Do not need to store messages • Each message can be resent to all the clients • New clients cannot access the chat history irc1.10gen.cc / 6667 / #fun

Slide 9

Slide 9 text

Message bus required to connect multiple servers Client Client Client Server 1 Server 2 Client Message Bus IRC Service !!! irc1.10gen.cc / 6667 / #fun

Slide 10

Slide 10 text

Capped collection as bus Client Client Client Server 1 Server 2 Client IRC Service Message pushed onto bus Server reads message from bus Client sends a message Message forwarded to clients in room Message forwarded to clients in room irc1.10gen.cc / 6667 / #fun

Slide 11

Slide 11 text

Message Bus Code • Add a message • Retrieve a message • Why was it a good idea to use MongoDB? irc1.10gen.cc / 6667 / #fun

Slide 12

Slide 12 text

Building an IRC service with MongoDB irc1.10gen.cc / 6667 / #fun Github: erh/mongo-irc Eliot Horowitz, CTO Jared Rosoff, Product Marketing Edouard Servan-Schreiber, Solution Architecture

Slide 13

Slide 13 text

Slide 14

Slide 14 text

What is IRC? irc1.10gen.cc / 6667 / #fun

Slide 15

Slide 15 text

Message bus required to connect multiple servers Client Client Client Server 1 Server 2 Client Message Bus IRC Service !!! irc1.10gen.cc / 6667 / #fun

Slide 16

Slide 16 text

In this talk • Upgrading from single node to replica set • Setting up MMS • Mongostat() • Delayed and hidden replication • Breakdowns • Secondary down • Primary down • Oplog runs out irc1.10gen.cc / 6667 / #fun

Slide 17

Slide 17 text

Development mode • Run everything on one node • Easy to code, test, repeat • But you don’t want to run in production this way Virtual Machine JVM MongoDB irc1.10gen.cc / 6667 / #fun

Slide 18

Slide 18 text

First deployment Virtual Machine JVM MongoD JVM MongoD MongoD MongoD Replica Set irc1.10gen.cc / 6667 / #fun

Slide 19

Slide 19 text

Servers – Single Server irc1 irc2 irc3 irc4 irc5 irc6 irc7 irc8 irc9 irc10 irc11 irc12 ircd mongod irc1.10gen.cc / 6667 / #fun

Slide 20

Slide 20 text

Servers – Replica Set irc1 irc2 irc3 irc4 irc5 irc6 irc7 irc8 irc9 irc10 irc11 irc12 ircd ircd mongod mongod mongod irc1.10gen.cc / 6667 / #fun

Slide 21

Slide 21 text

Single Node Replica Set • Spinup irc2.10gen.cc, irc3.10gen.cc • Mongod –replset irc –fork –v –logpath = … • RS.initiate() • RS.add(“irc2:27017”) • RS.add(“irc3:27017”) irc1.10gen.cc / 6667 / #fun

Slide 22

Slide 22 text

Mongostat Counters Memory Concurrency Network Replication irc1.10gen.cc / 6667 / #fun

Slide 23

Slide 23 text

MMS - Mongo Monitoring Service • Setup Python • python agent.py • nohup python agent.py > /[LOG-DIRECTORY]/ agent.log 2>&1 & • Setup log upload irc1.10gen.cc / 6667 / #fun

Slide 24

Slide 24 text

Add a delayed slave JVM MongoD MongoD MongoD Delayed Slave Delayed slave gives you time to recover from a human error “drop collection…” Otherwise all errors are replicated on the spot…. RS.add( “irc10:27017”, {slaveDelay: 900} ) irc1.10gen.cc / 6667 / #fun

Slide 25

Slide 25 text

Remote Hidden JVM MongoD MongoD MongoD Delayed Slave Remote Data Center Hidden Slave A hidden replica adds DR protection in case your data center crashes... RS.add( “irc11:27017”, {hidden: true} ) irc1.10gen.cc / 6667 / #fun

Slide 26

Slide 26 text

Oplog 1 2 3 4 5 6 7 8 9 10 11 12 13 Oplog T1 T2 T3 T4 Clients Read Operation Write Operation Time irc1.10gen.cc / 6667 / #fun

Slide 27

Slide 27 text

Oplog • Capped collection • Size determines how many operations fit • Recovery is faster when last operation is more recent than tail of primary’s oplog Head Tail t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15 t16 irc1.10gen.cc / 6667 / #fun

Slide 28

Slide 28 text

Oplog Head Tail t17 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15 t16 • Capped collection • Size determines how many operations fit • Recovery is faster when last operation is more recent than tail of primary’s oplog irc1.10gen.cc / 6667 / #fun

Slide 29

Slide 29 text

Oplog Head Tail t17 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15 t16 t18 • Capped collection • Size determines how many operations fit • Recovery is faster when last operation is more recent than tail of primary’s oplog irc1.10gen.cc / 6667 / #fun

Slide 30

Slide 30 text

Oplog Head Tail t17 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15 t16 t18 t19 • Capped collection • Size determines how many operations fit • Recovery is faster when last operation is more recent than tail of primary’s oplog irc1.10gen.cc / 6667 / #fun

Slide 31

Slide 31 text

Let’s Break Things…. • Oplog runs out on replica • Resync OR restore recent backup MongoD MongoD MongoD irc1.10gen.cc / 6667 / #fun

Slide 32

Slide 32 text

Building an IRC service with MongoDB irc1.10gen.cc / 6667 / #fun Github: erh/mongo-irc Eliot Horowitz, CTO Jared Rosoff, Product Marketing Edouard Servan-Schreiber, Solution Architecture

Slide 33

Slide 33 text

Slide 34

Slide 34 text

What is IRC? irc1.10gen.cc / 6667 / #fun

Slide 35

Slide 35 text

First deployment Virtual Machine JVM MongoD JVM MongoD MongoD MongoD Replica Set irc1.10gen.cc / 6667 / #fun

Slide 36

Slide 36 text

In this talk • Schema Design for the Log History • From Replica Set to Sharded Cluster irc1.10gen.cc / 6667 / #fun

Slide 37

Slide 37 text

How should we model the Chat History? Relational Way – One Doc per Message { _id : ObjectId() , room : , user : , time : , msg : “I believe third normal form is God…” } • Pros • Elegant • Documents do not grow • Inserts are simple • Cons • Querying per room or per user is poor performance irc1.10gen.cc / 6667 / #fun

Slide 38

Slide 38 text

How should we model the Chat History? One Doc per Room { _id : ObjectId() , room : , chat_history : [ { user : no3nf , time : ‘5/4/2012 9:01’, msg : “Down with 3NF!” } , { user : yes3nf , time : ‘5/4/2012 9:00’ , msg : “Long live 3NF!” } , …. ] } • Pros • Conceptually simple • Fast • Room traffic easy to isolate • Cons • Documents can grow with no bounds • 16MB hard limit • Appending to a long list takes a long time • Querying per user is non- trivial • Is that bad? Not necessarily irc1.10gen.cc / 6667 / #fun

Slide 39

Slide 39 text

How should we model the Chat History? Bucketed Approach – by size • One doc per room/bucket of 100 msgs { _id : @@@ , logs: [ { ts: ‘5/4/2012 9:01’, room: msg : “@ Down with 3NF!” } , { ts: ‘5/4/2012 9:02’, room: msg : “@ Long live 3NF!” } , { ts: ‘5/4/2012 9:03’, room: msg : “@ What am I doing here…” }, …. ] } • Message count maintained in the room object • Lookup the most recent bucket with div 10 • Bucket of 10 Messages • A bucket can span several days irc1.10gen.cc / 6667 / #fun

Slide 40

Slide 40 text

One Doc per Room/Bucket of K Msgs • Pros • Recent room traffic easy to isolate and access • Documents have a max size modulo the variation in message length • Traffic spikes and lulls have no impact • All room traffic easy to assemble • Cons • Requires preallocation to avoid impact of growing documents • Inserting new messages is a little harder irc1.10gen.cc / 6667 / #fun

Slide 41

Slide 41 text

Indexes • What will you need? • Look for room traffic • Identify the most recent bucket • Assemble the overall traffic • Default Index • _id = (room_id, bucket_id) irc1.10gen.cc / 6667 / #fun

Slide 42

Slide 42 text

Servers – Replica Set irc1 irc2 irc3 irc4 irc5 irc6 irc7 irc8 irc9 irc10 irc11 irc12 ircd ircd mongod mongod mongod irc1.10gen.cc / 6667 / #fun

Slide 43

Slide 43 text

Servers – Shards irc1 irc2 irc3 irc4 irc5 irc6 irc7 irc8 irc9 irc10 irc11 irc12 ircd mongos ircd mongos S1 mongod config config config S1 mongod S1 mongod S2 mongod S2 mongod S2 mongod S3 mongod S3 mongod S3 mongod irc1.10gen.cc / 6667 / #fun

Slide 44

Slide 44 text

Sharding client mongos ... mongos mongod mongod ... Shards mongod mongod mongod Config Servers mongod mongod mongod mongod mongod mongod mongod client client client irc1.10gen.cc / 6667 / #fun

Slide 45

Slide 45 text

Building an IRC service with MongoDB irc1.10gen.cc / 6667 / #fun Github: erh/mongo-irc Eliot Horowitz, CTO Jared Rosoff, Product Marketing Edouard Servan-Schreiber, Solution Architecture

Slide 46

Slide 46 text

Slide 47

Slide 47 text

What is IRC? irc1.10gen.cc / 6667 / #fun

Slide 48

Slide 48 text

Slide 49

Slide 49 text

In this talk • Performing Backups • Recovery • MMS usage • Chunk splits • Chunk migration irc1.10gen.cc / 6667 / #fun

Slide 50

Slide 50 text

Database Instance Backups { x: 1 } { x: 1 } { x: 1 } { x: 1 } { x: 1 } { x: 1 } { x: 1 } { x: 1 } Data Files Journal Files Consistent Snapshot MongoDump MongoExport fsync + lock

Slide 51

Slide 51 text

Backing up a Replica Set Database Instance Data Files Journal Files Database Instance Data Files Journal Files Database Instance Data Files Journal Files Primary Secondary Secondary Snapshot

Slide 52

Slide 52 text

Backing up a Sharded Cluster Not that much different than backing up a replica set. 1. Stop balancer 2. Back up each shard 3. Backup one config server 4. Turn balancer back on irc1.10gen.cc / 6667 / #fun

Slide 53

Slide 53 text

Restore from Backup • Mongodump/Mongoimport • data only, no indices • Must rebuild indices • Copy data files from snapshot • Data and indices – faster time to recovery • Priming a replica from a backup to help catchup irc1.10gen.cc / 6667 / #fun

Slide 54

Slide 54 text

Config Server Failure client mongos ... mongos mongod mongod ... Shards mongod mongod mongod Config Servers mongod mongod mongod mongod mongod mongod mongod client client client • Config becomes read only • Mongo can still read and write • But cannot split or move chunks irc1.10gen.cc / 6667 / #fun

Slide 55

Slide 55 text

MMS MongoD MongoD MongoD MMS Agent MMS Server irc1.10gen.cc / 6667 / #fun

Slide 56

Slide 56 text

Monitoring - Basics What to monitor What it tells you Operation Counters • How many requests is your database serving? Memory Utilization • How big is your dataset? • How much of it is resident in memory? • How much free memory do you have Page Faults • How often are page faults occuring • Does your data set fit in memory? Index Miss • How often are index scans hitting pages on disk? • Do your indexes fit in memory? Queue Depths • How often are clients waiting to run? • Is there lock contention? irc1.10gen.cc / 6667 / #fun

Slide 57

Slide 57 text

Monitoring - MMS irc1.10gen.cc / 6667 / #fun

Slide 58

Slide 58 text

MMS MongoD MongoD MongoD MMS Agent MMS Server Runs on your servers. Discovers your mongo server instances & collects stats. Data reported over secure HTTPS connection to MMS Service Operated by 10gen. Nothing for you to set up other than agent installation Secure login. Supports multiple users / deployments. irc1.10gen.cc / 6667 / #fun

Slide 59

Slide 59 text

Machine Statistics What to monitor What it tells you CPU • User, System, IOWait, Steal • Load average & what it actually is Disk • iops • io latency • What io %util really means Memory • free –m irc1.10gen.cc / 6667 / #fun

Slide 60

Slide 60 text

Summary • Built IRC server on Mongo • Capped collection for Message Bus • Single Node to Replica Set • Scalable Message Log • Replica Set to Sharded Cluster • Backup methods • Monitoring irc1.10gen.cc / 6667 / #fun

Slide 61

Slide 61 text

Building an IRC service with MongoDB irc1.10gen.cc / 6667 / #fun Github: erh/mongo-irc Eliot Horowitz, CTO Jared Rosoff, Product Marketing Edouard Servan-Schreiber, Solution Architecture