Upgrade to Pro — share decks privately, control downloads, hide ads and more …

MongoNYC 2012: Building a MongoDB Power Chat Server

mongodb
May 29, 2012
3.1k

MongoNYC 2012: Building a MongoDB Power Chat Server

MongoNYC 2012: Building a MongoDB Power Chat Server, Eliot Horowitz, Jared Rosoff, & Edouard Servan-Schreiber, 10gen. We will build an IRC server based on MongoDB.

mongodb

May 29, 2012
Tweet

Transcript

  1. Building an IRC service with MongoDB irc1.10gen.cc / 6667 /

    #fun Github: erh/mongo-irc Eliot Horowitz, CTO Jared Rosoff, Product Marketing Edouard Servan-Schreiber, Solution Architecture
  2. • 10:00 – 10:40 App Design, Schema Design, MessageBus on

    MongoDB • 10:45 – 11:25 Deployment, Replication, Monitoring • 11:40 – 12:20 Scaling Message Logging and Sharding – Live! • 12:25 – 1:05 Backups, Monitoring and Operations for a Sharded Cluster Agenda irc1.10gen.cc / 6667 / #fun
  3. In this talk • We’re building an IRC server •

    Architectural overview • Data model overview • Designing a Message Bus • Demo the app irc1.10gen.cc / 6667 / #fun
  4. Schema Design • Rooms { "_id" : "#fun", "topic" :

    "not set", "created" : ISODate("2012-05-21T19:53:42.690Z") } • Servers { "_id" : "ip-10-190-215-182:6667", "last_ping" : ISODate("2012-05-21T20:15:55.379Z") } irc1.10gen.cc / 6667 / #fun
  5. MessageBus { "_id" : ISODate("2012-05-21T19:53:42.691Z"), "type" : "room", "room" :

    "#fun", "message" : ":edouardss!~edouardss@0 JOIN #fun", "incarnation" : "Mon May 21 19:52:03 UTC 2012-0.7612059410121808" } { "_id" : ISODate("2012-05-22T18:29:15.950Z"), "type" : "room", "room" : "#fun", "message" : ":edouardss!~edouardss@0 PRIVMSG #fun :hello mongonyc", "incarnation" : "Tue May 22 18:19:25 UTC 2012-0.5635396421023062" } irc1.10gen.cc / 6667 / #fun
  6. Data for Single Server Setup • Keep track of rooms

    • Do not need to store messages • Each message can be resent to all the clients • New clients cannot access the chat history irc1.10gen.cc / 6667 / #fun
  7. Message bus required to connect multiple servers Client Client Client

    Server 1 Server 2 Client Message Bus IRC Service !!! irc1.10gen.cc / 6667 / #fun
  8. Capped collection as bus Client Client Client Server 1 Server

    2 Client IRC Service Message pushed onto bus Server reads message from bus Client sends a message Message forwarded to clients in room Message forwarded to clients in room irc1.10gen.cc / 6667 / #fun
  9. Message Bus Code • Add a message • Retrieve a

    message • Why was it a good idea to use MongoDB? irc1.10gen.cc / 6667 / #fun
  10. Building an IRC service with MongoDB irc1.10gen.cc / 6667 /

    #fun Github: erh/mongo-irc Eliot Horowitz, CTO Jared Rosoff, Product Marketing Edouard Servan-Schreiber, Solution Architecture
  11. • 10:00 – 10:40 App Design, Schema Design, MessageBus on

    MongoDB • 10:45 – 11:25 Deployment, Replication, Monitoring • 11:40 – 12:20 Scaling Message Logging and Sharding – Live! • 12:25 – 1:05 Backups, Monitoring and Operations for a Sharded Cluster Agenda irc1.10gen.cc / 6667 / #fun
  12. Message bus required to connect multiple servers Client Client Client

    Server 1 Server 2 Client Message Bus IRC Service !!! irc1.10gen.cc / 6667 / #fun
  13. In this talk • Upgrading from single node to replica

    set • Setting up MMS • Mongostat() • Delayed and hidden replication • Breakdowns • Secondary down • Primary down • Oplog runs out irc1.10gen.cc / 6667 / #fun
  14. Development mode • Run everything on one node • Easy

    to code, test, repeat • But you don’t want to run in production this way Virtual Machine JVM MongoDB irc1.10gen.cc / 6667 / #fun
  15. Servers – Single Server irc1 irc2 irc3 irc4 irc5 irc6

    irc7 irc8 irc9 irc10 irc11 irc12 ircd mongod irc1.10gen.cc / 6667 / #fun
  16. Servers – Replica Set irc1 irc2 irc3 irc4 irc5 irc6

    irc7 irc8 irc9 irc10 irc11 irc12 ircd ircd mongod mongod mongod irc1.10gen.cc / 6667 / #fun
  17. Single Node Replica Set • Spinup irc2.10gen.cc, irc3.10gen.cc • Mongod

    –replset irc –fork –v –logpath = … • RS.initiate() • RS.add(“irc2:27017”) • RS.add(“irc3:27017”) irc1.10gen.cc / 6667 / #fun
  18. MMS - Mongo Monitoring Service • Setup Python • python

    agent.py • nohup python agent.py > /[LOG-DIRECTORY]/ agent.log 2>&1 & • Setup log upload irc1.10gen.cc / 6667 / #fun
  19. Add a delayed slave JVM MongoD MongoD MongoD Delayed Slave

    Delayed slave gives you time to recover from a human error “drop collection…” Otherwise all errors are replicated on the spot…. RS.add( “irc10:27017”, {slaveDelay: 900} ) irc1.10gen.cc / 6667 / #fun
  20. Remote Hidden JVM MongoD MongoD MongoD Delayed Slave Remote Data

    Center Hidden Slave A hidden replica adds DR protection in case your data center crashes... RS.add( “irc11:27017”, {hidden: true} ) irc1.10gen.cc / 6667 / #fun
  21. Oplog 1 2 3 4 5 6 7 8 9

    10 11 12 13 Oplog T1 T2 T3 T4 Clients Read Operation Write Operation Time irc1.10gen.cc / 6667 / #fun
  22. Oplog • Capped collection • Size determines how many operations

    fit • Recovery is faster when last operation is more recent than tail of primary’s oplog Head Tail t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15 t16 irc1.10gen.cc / 6667 / #fun
  23. Oplog Head Tail t17 t2 t3 t4 t5 t6 t7

    t8 t9 t10 t11 t12 t13 t14 t15 t16 • Capped collection • Size determines how many operations fit • Recovery is faster when last operation is more recent than tail of primary’s oplog irc1.10gen.cc / 6667 / #fun
  24. Oplog Head Tail t17 t3 t4 t5 t6 t7 t8

    t9 t10 t11 t12 t13 t14 t15 t16 t18 • Capped collection • Size determines how many operations fit • Recovery is faster when last operation is more recent than tail of primary’s oplog irc1.10gen.cc / 6667 / #fun
  25. Oplog Head Tail t17 t4 t5 t6 t7 t8 t9

    t10 t11 t12 t13 t14 t15 t16 t18 t19 • Capped collection • Size determines how many operations fit • Recovery is faster when last operation is more recent than tail of primary’s oplog irc1.10gen.cc / 6667 / #fun
  26. Let’s Break Things…. • Oplog runs out on replica •

    Resync OR restore recent backup MongoD MongoD MongoD irc1.10gen.cc / 6667 / #fun
  27. Building an IRC service with MongoDB irc1.10gen.cc / 6667 /

    #fun Github: erh/mongo-irc Eliot Horowitz, CTO Jared Rosoff, Product Marketing Edouard Servan-Schreiber, Solution Architecture
  28. • 10:00 – 10:40 App Design, Schema Design, MessageBus on

    MongoDB • 10:45 – 11:25 Deployment, Replication, Monitoring • 11:40 – 12:20 Scaling Message Logging and Sharding – Live! • 12:25 – 1:05 Backups, Monitoring and Operations for a Sharded Cluster Agenda irc1.10gen.cc / 6667 / #fun
  29. In this talk • Schema Design for the Log History

    • From Replica Set to Sharded Cluster irc1.10gen.cc / 6667 / #fun
  30. How should we model the Chat History? Relational Way –

    One Doc per Message { _id : ObjectId() , room : <roomid> , user : <userid> , time : <timestamp> , msg : “I believe third normal form is God…” } • Pros • Elegant • Documents do not grow • Inserts are simple • Cons • Querying per room or per user is poor performance irc1.10gen.cc / 6667 / #fun
  31. How should we model the Chat History? One Doc per

    Room { _id : ObjectId() , room : <roomid> , chat_history : [ { user : no3nf , time : ‘5/4/2012 9:01’, msg : “Down with 3NF!” } , { user : yes3nf , time : ‘5/4/2012 9:00’ , msg : “Long live 3NF!” } , …. ] } • Pros • Conceptually simple • Fast • Room traffic easy to isolate • Cons • Documents can grow with no bounds • 16MB hard limit • Appending to a long list takes a long time • Querying per user is non- trivial • Is that bad? Not necessarily irc1.10gen.cc / 6667 / #fun
  32. How should we model the Chat History? Bucketed Approach –

    by size • One doc per room/bucket of 100 msgs { _id : <room_id>@@@<bucket-id> , logs: [ { ts: ‘5/4/2012 9:01’, room: <room_id> msg : “<userid>@ <room_id> Down with 3NF!” } , { ts: ‘5/4/2012 9:02’, room: <room_id> msg : “<userid>@ <room_id> Long live 3NF!” } , { ts: ‘5/4/2012 9:03’, room: <room_id> msg : “<userid>@ <room_id> What am I doing here…” }, …. ] } • Message count maintained in the room object • Lookup the most recent bucket with <msg_count> div 10 • Bucket of 10 Messages • A bucket can span several days irc1.10gen.cc / 6667 / #fun
  33. One Doc per Room/Bucket of K Msgs • Pros •

    Recent room traffic easy to isolate and access • Documents have a max size modulo the variation in message length • Traffic spikes and lulls have no impact • All room traffic easy to assemble • Cons • Requires preallocation to avoid impact of growing documents • Inserting new messages is a little harder irc1.10gen.cc / 6667 / #fun
  34. Indexes • What will you need? • Look for room

    traffic • Identify the most recent bucket • Assemble the overall traffic • Default Index • _id = (room_id, bucket_id) irc1.10gen.cc / 6667 / #fun
  35. Servers – Replica Set irc1 irc2 irc3 irc4 irc5 irc6

    irc7 irc8 irc9 irc10 irc11 irc12 ircd ircd mongod mongod mongod irc1.10gen.cc / 6667 / #fun
  36. Servers – Shards irc1 irc2 irc3 irc4 irc5 irc6 irc7

    irc8 irc9 irc10 irc11 irc12 ircd mongos ircd mongos S1 mongod config config config S1 mongod S1 mongod S2 mongod S2 mongod S2 mongod S3 mongod S3 mongod S3 mongod irc1.10gen.cc / 6667 / #fun
  37. Sharding client mongos ... mongos mongod mongod ... Shards mongod

    mongod mongod Config Servers mongod mongod mongod mongod mongod mongod mongod client client client irc1.10gen.cc / 6667 / #fun
  38. Building an IRC service with MongoDB irc1.10gen.cc / 6667 /

    #fun Github: erh/mongo-irc Eliot Horowitz, CTO Jared Rosoff, Product Marketing Edouard Servan-Schreiber, Solution Architecture
  39. • 10:00 – 10:40 App Design, Schema Design, MessageBus on

    MongoDB • 10:45 – 11:25 Deployment, Replication, Monitoring • 11:40 – 12:20 Scaling Message Logging and Sharding – Live! • 12:25 – 1:05 Backups, Monitoring and Operations for a Sharded Cluster Agenda irc1.10gen.cc / 6667 / #fun
  40. Servers – Shards irc1 irc2 irc3 irc4 irc5 irc6 irc7

    irc8 irc9 irc10 irc11 irc12 ircd mongos ircd mongos S1 mongod config config config S1 mongod S1 mongod S2 mongod S2 mongod S2 mongod S3 mongod S3 mongod S3 mongod irc1.10gen.cc / 6667 / #fun
  41. In this talk • Performing Backups • Recovery • MMS

    usage • Chunk splits • Chunk migration irc1.10gen.cc / 6667 / #fun
  42. Database Instance Backups { x: 1 } { x: 1

    } { x: 1 } { x: 1 } { x: 1 } { x: 1 } { x: 1 } { x: 1 } Data Files Journal Files Consistent Snapshot MongoDump MongoExport fsync + lock
  43. Backing up a Replica Set Database Instance Data Files Journal

    Files Database Instance Data Files Journal Files Database Instance Data Files Journal Files Primary Secondary Secondary Snapshot
  44. Backing up a Sharded Cluster Not that much different than

    backing up a replica set. 1. Stop balancer 2. Back up each shard 3. Backup one config server 4. Turn balancer back on irc1.10gen.cc / 6667 / #fun
  45. Restore from Backup • Mongodump/Mongoimport • data only, no indices

    • Must rebuild indices • Copy data files from snapshot • Data and indices – faster time to recovery • Priming a replica from a backup to help catchup irc1.10gen.cc / 6667 / #fun
  46. Config Server Failure client mongos ... mongos mongod mongod ...

    Shards mongod mongod mongod Config Servers mongod mongod mongod mongod mongod mongod mongod client client client • Config becomes read only • Mongo can still read and write • But cannot split or move chunks irc1.10gen.cc / 6667 / #fun
  47. Monitoring - Basics What to monitor What it tells you

    Operation Counters • How many requests is your database serving? Memory Utilization • How big is your dataset? • How much of it is resident in memory? • How much free memory do you have Page Faults • How often are page faults occuring • Does your data set fit in memory? Index Miss • How often are index scans hitting pages on disk? • Do your indexes fit in memory? Queue Depths • How often are clients waiting to run? • Is there lock contention? irc1.10gen.cc / 6667 / #fun
  48. MMS MongoD MongoD MongoD MMS Agent MMS Server Runs on

    your servers. Discovers your mongo server instances & collects stats. Data reported over secure HTTPS connection to MMS Service Operated by 10gen. Nothing for you to set up other than agent installation Secure login. Supports multiple users / deployments. irc1.10gen.cc / 6667 / #fun
  49. Machine Statistics What to monitor What it tells you CPU

    • User, System, IOWait, Steal • Load average & what it actually is Disk • iops • io latency • What io %util really means Memory • free –m irc1.10gen.cc / 6667 / #fun
  50. Summary • Built IRC server on Mongo • Capped collection

    for Message Bus • Single Node to Replica Set • Scalable Message Log • Replica Set to Sharded Cluster • Backup methods • Monitoring irc1.10gen.cc / 6667 / #fun
  51. Building an IRC service with MongoDB irc1.10gen.cc / 6667 /

    #fun Github: erh/mongo-irc Eliot Horowitz, CTO Jared Rosoff, Product Marketing Edouard Servan-Schreiber, Solution Architecture