MongoDB Off The Web - Mongo Bristol

Some thoughts on using MongoDB for things other than Web Applications.

Transcript

1. Mongo Bristol Meetup “MongoDB Off The Web” Some Thoughts on Using Mongo for ‘Other’ Things Brendan McAdams - 10gen, Inc. brendan@10gen.com @rit Monday, July 4, 2011
2. Introductions • Brendan McAdams <brendan@10gen.com> • Started using MongoDB (in production) ~Feb. 2009 • Engineer at 10gen - “The Company” • Scala support (maintain and develop drivers, improve and assist third party frameworks, community steering) • Java support (contribute to maintenance of drivers, features. Focus is on improving integration for non-Java JVM languages w/ our Java toolchain) • Hadoop support (develop & maintain MongoDB’s Hadoop integration layers, assist deployments) • Support (Free community & commercial) • Community Outreach (Meetups, Conferences) • Training & Consulting Monday, July 4, 2011
3. MongoDB is fast, light and flexible It’s use of JSON gives it a natural affinity to the Web. Monday, July 4, 2011
4. MongoDB is fast, light and flexible It’s use of JSON gives it a natural affinity to the Web. It has other features that make it perfect for “other” things too. Monday, July 4, 2011
5. Some Solid “non-web” Use Patterns • Event / Pipeline Processing • Logging • Graylog2 • Flume Sink • Durable Messaging • Broadcast Messaging • Pub/Sub Monday, July 4, 2011
6. Today, Let’s Discuss Messaging Monday, July 4, 2011
7. Today, Let’s Discuss Messaging • Messaging is something that, with MongoDB, you “roll your own” Monday, July 4, 2011
8. Today, Let’s Discuss Messaging • Messaging is something that, with MongoDB, you “roll your own” • But there are great builtin facilities to make it easier for you to do it Monday, July 4, 2011
9. Today, Let’s Discuss Messaging • Messaging is something that, with MongoDB, you “roll your own” • But there are great builtin facilities to make it easier for you to do it • Capped Collections Monday, July 4, 2011
10. Today, Let’s Discuss Messaging • Messaging is something that, with MongoDB, you “roll your own” • But there are great builtin facilities to make it easier for you to do it • Capped Collections • Tailable cursors Monday, July 4, 2011
11. Today, Let’s Discuss Messaging • Messaging is something that, with MongoDB, you “roll your own” • But there are great builtin facilities to make it easier for you to do it • Capped Collections • Tailable cursors • findAndModify Monday, July 4, 2011
12. WTF Is A Capped Collection? Monday, July 4, 2011
13. WTF Is A Capped Collection? • Special size bounded MongoDB collection designed for Replication Monday, July 4, 2011
14. WTF Is A Capped Collection? • Special size bounded MongoDB collection designed for Replication • Created specially with a number of bytes it may hold Monday, July 4, 2011
15. WTF Is A Capped Collection? • Special size bounded MongoDB collection designed for Replication • Created specially with a number of bytes it may hold • No _id index Monday, July 4, 2011
16. WTF Is A Capped Collection? • Special size bounded MongoDB collection designed for Replication • Created specially with a number of bytes it may hold • No _id index • Documents are maintained in insertion order Monday, July 4, 2011
17. WTF Is A Capped Collection? • Special size bounded MongoDB collection designed for Replication • Created specially with a number of bytes it may hold • No _id index • Documents are maintained in insertion order • No deletes allowed Monday, July 4, 2011
18. WTF Is A Capped Collection? • Special size bounded MongoDB collection designed for Replication • Created specially with a number of bytes it may hold • No _id index • Documents are maintained in insertion order • No deletes allowed • Updates only allowed if document won’t “grow” Monday, July 4, 2011
19. WTF Is A Capped Collection? • Special size bounded MongoDB collection designed for Replication • Created specially with a number of bytes it may hold • No _id index • Documents are maintained in insertion order • No deletes allowed • Updates only allowed if document won’t “grow” • As collection fills up, oldest entries “fall out” Monday, July 4, 2011
20. WTF Is A Capped Collection? • Special size bounded MongoDB collection designed for Replication • Created specially with a number of bytes it may hold • No _id index • Documents are maintained in insertion order • No deletes allowed • Updates only allowed if document won’t “grow” • As collection fills up, oldest entries “fall out” • Allow for a special cursor type: Tailable Cursors Monday, July 4, 2011
21. tail -f `mongo ‘db.data.find()’` Monday, July 4, 2011
22. tail -f `mongo ‘db.data.find()’` • Tailable Cursors mode are a special cursor mode in MongoDB Monday, July 4, 2011
23. tail -f `mongo ‘db.data.find()’` • Tailable Cursors mode are a special cursor mode in MongoDB • Similar to Unix’ ‘tail -f’, maintain a pointer to the last document seen; continue moving forward as new documents added Monday, July 4, 2011
24. tail -f `mongo ‘db.data.find()’` • Tailable Cursors mode are a special cursor mode in MongoDB • Similar to Unix’ ‘tail -f’, maintain a pointer to the last document seen; continue moving forward as new documents added • With “Await” cursor mode, can poll until new documents arrive Monday, July 4, 2011
25. tail -f `mongo ‘db.data.find()’` • Tailable Cursors mode are a special cursor mode in MongoDB • Similar to Unix’ ‘tail -f’, maintain a pointer to the last document seen; continue moving forward as new documents added • With “Await” cursor mode, can poll until new documents arrive • Incredibly efficient for non-indexed queries Monday, July 4, 2011
26. Broadcast Messaging Made Easy Monday, July 4, 2011
27. Broadcast Messaging Made Easy • This mechanism allows for very easy broadcast messaging Monday, July 4, 2011
28. Broadcast Messaging Made Easy • This mechanism allows for very easy broadcast messaging • ... In fact, it is exactly how MongoDB does replication Monday, July 4, 2011
29. Broadcast Messaging Made Easy • This mechanism allows for very easy broadcast messaging • ... In fact, it is exactly how MongoDB does replication • Because you can’t delete messages this wouldn’t be ideal for pub/sub Monday, July 4, 2011
30. Broadcast Messaging Made Easy • This mechanism allows for very easy broadcast messaging • ... In fact, it is exactly how MongoDB does replication • Because you can’t delete messages this wouldn’t be ideal for pub/sub • But could be paired carefully with findAndModify Monday, July 4, 2011
31. Pub/Sub and findAndModify Monday, July 4, 2011
32. Pub/Sub and findAndModify • Compare and Swap / ABA Problems can be tricky • AKA “Distributed Locking is Hard - Let’s Go Shopping!” • MongoDB’s update doesn’t allow you to fetch the exact document(s) changed • The findAndModify command enables a proper mechanism • Find and modify first matching document and return new doc or old one • Find and remove first matching document and return the pre-removed document • Isolated; two competing threads won’t get the same document Monday, July 4, 2011
33. Lots of Ideas to be Explored • Akka (Scala distributed computing & Actor framework) will include a MongoDB based durable mailbox soon, using these concepts for bounded and unbounded messaging • 10gen’s MMS monitoring service uses findAndModify to facilitate worker queues Monday, July 4, 2011
34. @mongodb conferences,  appearances,  and  meetups http://www.10gen.com/events http://bit.ly/mongo>   Facebook                    |                  Twitter                  |                  LinkedIn http://linkd.in/joinmongo download at mongodb.org We’re Hiring ! brendan@10gen.com (twitter: @rit) Monday, July 4, 2011