Rivieria Scala and Clojure - MongoDB talk

My MongoDB talk from the recent Rivieria Scala and Clojure usergroup.

http://www.meetup.com/riviera-scala-clojure/events/89448102/

Transcript

1. Riviera - Scala Clojure Ross Lawley Software Engineer, 10gen @RossC0
2. 10gen and MongoDB
3. 3 10gen Overview 10gen is the company behind MongoDB – the leading NoSQL database
4. 4 10gen Overview Offices in New York, Palo Alto, Washington DC, London, Dublin, Barcelona and Sydney
5. MongoDB Overview
6. Relational Database Challenges Data Types • Unstructured data • Semi-structured data • Polymorphic data Volume of Data • Petabytes of data • Trillions of records • Tens of millions of queries per second Agile Development • Iterative • Short development cycles • New workloads New Architectures • Horizontal scaling • Commodity servers • Cloud computing
7. Database Landscape Depth of Functionality Scalability & Performance Memcached MongoDB RDBMS
8. MongoDB is a ___________ database • Document-oriented • Open-source • High performance and horizontally scalable • Full featured
9. Document-Oriented Database • Not .PDF & .DOC files • A document is essentially an associative array – JSON object, PHP Array, Python Dictionary, etc. • BSON – www.bsonspec.org
10. Open-Source • MongoDB is an open-source project • On GitHub • Database licensed under the AGPL • Drivers licensed under Apache
11. High Performance and Horizontally Scalable • High performance – Written in C++ – Data serialised as BSON (fast parsing) – Full support for primary & secondary indexes • Horizontally scalable – Auto-sharding – Scale across • Commodity hardware • Cloud compute • Hybrid
12. Full Featured • Rich Ad Hoc queries • Real time aggregation • Geospatial features • Native bindings for most programming languages
13. MongoDB is a Single-Master System • All writes are to a primary (master) • Failure of the primary is detected, and a new one is elected • Application writes get an error if there is no quorum to elect a new master – Reads can continue
14. MongoDB Storage Management • Data is kept in memory-mapped files • Files are allocated as needed • Indexes (B*-trees) point to documents using geographical addresses
15. Relational Schema User ·Name ·Email address Category ·Name ·URL Comment ·Comment ·Date ·Author Article ·Name ·Slug ·Publish date ·Text Tag ·Name ·URL
16. MongoDB Schema User ·Name ·Email address Article ·Name ·Slug ·Publish date ·Text ·Author Comment[] ·Comment ·Date ·Author Tag[] ·Value Category[] ·Value
17. Reliability – Replica Sets Secondary Secondary Primary Client Application Driver Write Read Read
18. Scalability – Auto-sharding Node 1 Secondary Config Server Node 1 Secondary Config Server Node 1 Secondary Config Server Shard Shard Shard Mongos App Server Mongos App Server Mongos App Server
19. Terminology RDBMS Mongo Table, View Collection Row(s) Document Index Index Join Linking/Embedding Partition Shard Partition Key Shard Key
20. Working with MongoDB
21. user = { username: "ross", first_name: "Ross", last_name: "Lawley"} > db.users.insert(user) Create (Insert)
22. > db.users.findOne() { "_id" : ObjectId("50ed3c5cab4ef39dc735664b"), "username" : "ross", "first_name" : "Ross", "last_name" : "Lawley" } Read (Query)
23. _id • _id is the primary key in MongoDB • Automatically indexed • Automatically created as an ObjectId if not provided • Any unique immutable value could be used
24. • ObjectId is a special 12 byte value • Guaranteed to be unique across your cluster ObjectId("50ed3c5cab4ef39dc735664b") |-------------||---------||-----||----------| ts mac pid inc ObjectId
25. // find users with any tags > db.users.find( {tags: {$exists: true }} ) // find users matching a regular expression > db.users.find( {username: /^ro*/i } ) // count posts by author > db.users.find( {username: "Ross"} ).count() • Conditional Operators – $all, $exists, $mod, $ne, $in, $nin, $nor, $or, $size, $type – $lt, $lte, $gt, $gte Query Operators
26. > tags = ["superuser", "db_admin"] > address = { street: "Scrutton Street", city: "London" } > db.users.update({}, {"$pushAll": {"tags": tags}, "$set": {"address": address}, "$inc": {"tag_count": 2}}) Update
27. > db.users.findOne() { "_id" : ObjectId("50ed3c5cab4ef39dc735664b"), "address" : { "street" : "Zetland House", "city" : "London" }, "first_name" : "Ross", "last_name" : "Lawley", "tag_count" : 2, "tags" : [ "superuser", "db_admin" ], "username" : "ross" } Read (Query)
28. • Scalar – $set, $unset, $inc, • Array – $push, $pushAll, $pull, $pullAll, $addToSet Atomic operators
29. // 1 means ascending, -1 means descending > db.users.ensureIndex({username: 1}) > db.users.find({username: "ross"}).explain() // Multi-key indexes > db.users.ensureIndex({tags: 1}) // index nested field > db.users.ensureIndex({"address.city": 1}) // Compound indexes > db.users.ensureIndex({ "username": 1, "address.city": 1 }) Secondary Indexes
30. MongoDB Drivers
31. Official 10Gen Drivers
32. MongoDB drivers • Official Support for 13 languages • Community drivers for tons more – Clojure, R, lua etc. • Drivers connect to mongo servers • Drivers translate BSON into native types • mongo shell is not a driver, but works like one in some ways • Installed using typical means (npm, pecl, gem, pip)
33. Scala
34. • Wraps the Java driver • Provide idiomatic Scala interface to MongoDB Scala
35. Scala case class Book(id: ObjectId, author: String, isbn: String, price: Price, year: Int, tags: Seq[String], title: String, publisher: String, edition: Option[String]) { def toDBObject = MongoDBObject( "_id" -> id, "author" -> author, "isbn" -> isbn, "price" -> price.toDBObject, "publicationYear" -> year, "tags" -> tags, "title" -> title, "publisher" -> publisher, "edition" -> edition ) }
36. Scala // Connect to default - localhost, 27017 scala> val mongoClient = MongoClient() mongoClient: com.mongodb.casbah.MongoClient ... // Chainable api scala> val mongoColl = mongoClient("bookstore")("books") mongoColl: com.mongodb.casbah.MongoCollection ...
37. Scala scala> val builder = MongoDBObject.newBuilder scala> builder += "foo" -> "bar" scala> builder += "x" -> "y" scala> builder += ("pie" -> 3.14) scala> builder += ("spam" -> "eggs", "mmm" -> "bacon") builder.type = com.mongodb.casbah.commons.MongoDBObjectBuilder // Return a DBObject scala> val newObj = builder.result newObj: com.mongodb.casbah.commons.Imports.DBObject = { "foo" : "bar" , "x" : "y" , "pie" : 3.14 , "spam" : "eggs" , "mmm" : "bacon"}
38. Scala scala> val newObj = MongoDBObject("foo" -> "bar", | "x" -> "y", | "pie" -> 3.14, | "spam" -> "eggs") newObj: com.mongodb.casbah.commons.Imports.DBObject = { "foo" : "bar" , "x" : "y" , "pie" : 3.14 , "spam" : "eggs"}
39. Scala val mongo: MongoClient = MongoClient()("bookstore")("books") def findAll() = for ( book <- mongo.find() ) yield new Book(book) findAll().foreach(b => println("<Book> " + b))
40. Scala val mongo: MongoClient = MongoClient()("bookstore")("books") val query: DBObject = ("price" $lt 40.00) ++ ("tag" -> "scala") mongo.find( query )
41. • Wraps the Java driver • Scala-esque way of interacting with mongo • Casbah 2.5.0 coming soon! Casbah
42. Salat
43. • Bi-directional Scala case class serialization library • Leverages MongoDB's DBObject (target format) • Depends on the latest releases of: -scalap -casbah-core -mongo-java-driver • https://github.com/novus/salat Salat
44. Salat - there and back case class Alpha(x: String) scala> val a = Alpha(x = "Hello world") a: com.novus.salat.test.model.Alpha = Alpha(Hello world) scala> val dbo = grater[Alpha].asDBObject(a) dbo: com.mongodb.casbah.Imports.DBObject = { "_typeHint" : "com.novus.salat.test.model.Alpha" , "x" : "Hello world"} scala> val a_* = grater[Alpha].asObject(dbo) a_*: com.novus.salat.test.model.Alpha = Alpha(Hello world) scala> a == a_* res0: Boolean = true
45. • A case class instance extends Scala's Product trait, which provides a product iterator over its elements. • Salat used pickled Scala signatures to turn case classes into indexed fields with associated type information. • These fields are then serialized or deserialized using the memoized indexed fields with type information. How does that work?
46. • Case classes • Case classes typed to a trait or an abstract superclass (requires @Salat) • Inside a case class constructor • Immutable collections: lists, seqs, maps whose key is a String • Options • Any type handled by BSON encoding hooks Salat - Supports
47. • classes • case classes nested inside an enclosing class or trait (Cake pattern - coming soon) • collection support needs to be improved - no Set, Array, mutable Map, etc Salat - Limitations
48. Simple way to get started and create your own DAO • insert and get back an Option with the id • findOne - get back an Option typed to your case class • find and get back a Mongo cursor typed to your class • iterate, limit, skip and sort • update with a query and a case class • save and remove case classes SalatDAO
49. SalatDAO - example import com.novus.salat._ import com.novus.salat.global._ case class Omega(_id: ObjectId = new ObjectId, z: String, y: Boolean) object OmegaDAO extends SalatDAO[Omega, ObjectId]( collection = MongoConnection()("salat-example")("omega"))
50. SalatDAO - Insert and find scala> val o = Omega(z = "something", y = false) o: Omega = Omega(4dac7b3e75e1b63949139c91, something, false) scala> val _id = OmegaDAO.insert(o) _id: Option[ObjectId] = Some(4dac7b3e75e1b63949139c91) scala> val o_* = OmegaDAO.findOne( MongoDBObject("z" -> "something")) o_*: Option[Omega] = Some( Omega(4dac7b3e75e1b63949139c91, something, false))
51. • Simplifies serialization to and from mongo • salatDAO provides a quick CRUD interface • New version coming soon! Salat
52. Scala - community drivers
53. • ReactiveMongo - asynchronous / non blocking by the guys at zenexity! Stephane Godbillon • Hammersmith - Netty + Akka.IO interfaces, strongly functional and callback based Scala - community drivers
54. • ReplicaSet support • Authentication support • GridFS support (streaming capable) • Cursors (providing a stream of documents) • Bulk inserts • Database commands support • Indexing operations ReactiveMongo
55. ReactiveMongo // select only my documents val query = BSONDocument("firstName" -> BSONString("Ross")) // get a Cursor[BSONDocument] val cursor = collection.find(query) //Enumerate this cursor and print a readable version cursor.enumerate.apply(Iteratee.foreach { doc => println("found document: " + DefaultBSONIterator.pretty(doc.bsonIterator)) })
56. ReactiveMongo // select only my documents val query = BSONDocument("firstName" -> BSONString("Ross")) // get a Cursor[BSONDocument] val cursor = collection.find(query) // Create a list using the toList helper val futurelist = cursor.toList futurelist.onSuccess { case list => val names = list.map( _.getAs[BSONString]("lastName").get.value) println("got names: " + names) }
57. Clojure - community drivers
58. http://clojuremongodb.info/ Monger
59. Monger (ns my.service.server (:require [monger.core :as mg]) (:import [com.mongodb MongoOptions ServerAddress])) ;; localhost, default port (mg/connect!) ;; set default database using set-db (mg/set-db! (mg/get-db "monger-test"))
60. Monger - Insert (connect!) (set-db! (monger.core/get-db "monger-test")) ;; with explicit document id (recommended) (insert "documents" { :_id (ObjectId.) :first_name "John" :last_name "Lennon" }) ;; with a different write concern (insert "documents" { :_id (ObjectId.) :first_name "John" :last_name "Lennon" } WriteConcern/JOURNAL_SAFE)
61. Monger - Querying (connect!) (set-db! (monger.core/get-db "monger-test")) ;; Query returns com.mongodb.DBObject object (mc/find "documents" {:first_name "Ringo"})
62. Monger - Querying (connect!) (set-db! (monger.core/get-db "monger-test")) ;; Query returns com.mongodb.DBObject object (mc/find "documents" {:first_name "Ringo"}) ;; returns documents as Clojure maps (mc/find-maps "documents" {:first_name "Ringo"})
63. • MongoDB is an agile, scalable NoSQL database • If you're coding on the JVM you have many options for interacting with MongoDB • Multiple native drivers • Some great community drivers In Summary
64. Click to edit Master text styles More Information Resource Location MongoDB Downloads www.mongodb.org/downloads Free Online Training education.10gen.com Webinars and Events www.10gen.com/events White Papers www.10gen.com/white-papers Customer Case Studies www.10gen.com/customers Presentations www.10gen.com/presentations Documentation docs.mongodb.org Additional Info info@10gen.com
65. • http://www.mongodb.org/display/DOCS/Drivers • http://reactivemongo.org/ • http://clojuremongodb.info/ More Information