NoSQL eXchange 2011: MongoDB + Scala: Case Classes, Documents and Shards for a New Data Model

Brendan McAdams -- creator of Casbah, a Scala toolkit for MongoDB -- will give a talk on "MongoDB + Scala: Case Classes, Documents and Shards for a New Data Model"

Transcript

1. MongoDB + Scala CASE CLASSES, DOCUMENTS AND SHARDS FOR A NEW DATA MODEL Brendan McAdams @rit 10gen, Inc. brendan@10gen.com Wednesday, November 2, 11
2. First, a bit of history (and some horrible metaphors) Wednesday, November 2, 11
3. First, a bit of history (and some horrible metaphors) Wednesday, November 2, 11
4. Meet my first love Wednesday, November 2, 11
5. Meet my first love Wednesday, November 2, 11
6. Your Datastore Should Serve Your Application's Architecture Wednesday, November 2, 11
7. … not Vice Versa Your Datastore Should Serve Your Application's Architecture Wednesday, November 2, 11
8. •Web Apps need lightweight, high performance, highly scalable data stores. •The cloud changes the expectations and performance profile of data stores. •Focus of the new "NoSQL" paradigm of lots of inexpensive cloud servers and horizontal scalability; slow disks and random failures not as huge an impact Data Access in the Right Context Wednesday, November 2, 11
9. What Is MongoDB? Wednesday, November 2, 11
10. So We’ve Built an Application with a Database Wednesday, November 2, 11
11. So We’ve Built an Application with a Database How do we integrate that database with our application’s object hierarchy? Wednesday, November 2, 11
12. I know! Let’s use an ORM! Wednesday, November 2, 11
13. I know! Let’s use an ORM! Congratulations: Now we’ve got 2 problems! (or is it n+1?) Wednesday, November 2, 11
14. Let’s Face It ... SQL Sucks. Wednesday, November 2, 11
15. Let’s Face It ... SQL Sucks. For some problems at least. Wednesday, November 2, 11
16. Stuffing an object graph into a relational model is like fitting a square peg into a round hole. Wednesday, November 2, 11
17. Sure, we can use an ORM. But who are we really fooling? Wednesday, November 2, 11
18. Sure, we can use an ORM. But who are we really fooling? ... and who/what are we going to wake up next to in the morning? Wednesday, November 2, 11
19. Make Your Data Work for You Wednesday, November 2, 11
20. •Object Graphs are often orthogonal to relational models. Make Your Data Work for You Wednesday, November 2, 11
21. •Object Graphs are often orthogonal to relational models. •Lifetimes of blood, sweat, tears and late night plots of homicide surround ORMS. They simply defer the pain (and make debugging harder) Make Your Data Work for You Wednesday, November 2, 11
22. •Object Graphs are often orthogonal to relational models. •Lifetimes of blood, sweat, tears and late night plots of homicide surround ORMS. They simply defer the pain (and make debugging harder) •… Is a generic query really the right answer for your app?) Make Your Data Work for You Wednesday, November 2, 11
23. This is a SQL Model mysql> select * from book; +----+----------------------------------------------------------+ | id | title | +----+----------------------------------------------------------+ | 1 | The Demon-Haunted World: Science as a Candle in the Dark | | 2 | Cosmos | | 3 | Programming in Scala | +----+----------------------------------------------------------+ 3 rows in set (0.00 sec) mysql> select * from bookauthor; +---------+-----------+ | book_id | author_id | +---------+-----------+ | 1 | 1 | | 2 | 1 | | 3 | 2 | | 3 | 3 | | 3 | 4 | +---------+-----------+ 5 rows in set (0.00 sec) mysql> select * from author; +----+-----------+------------+-------------+-------------+---------------+ | id | last_name | first_name | middle_name | nationality | year_of_birth | +----+-----------+------------+-------------+-------------+---------------+ | 1 | Sagan | Carl | Edward | NULL | 1934 | | 2 | Odersky | Martin | NULL | DE | 1958 | | 3 | Spoon | Lex | NULL | NULL | NULL | | 4 | Venners | Bill | NULL | NULL | NULL | +----+-----------+------------+-------------+-------------+---------------+ 4 rows in set (0.00 sec) Wednesday, November 2, 11
24. Joins are great and all ... Wednesday, November 2, 11
25. Joins are great and all ... • Potentially organizationally messy Wednesday, November 2, 11
26. Joins are great and all ... • Potentially organizationally messy • Structure of a single object is NOT immediately clear to someone glancing at the shell data Wednesday, November 2, 11
27. Joins are great and all ... • Potentially organizationally messy • Structure of a single object is NOT immediately clear to someone glancing at the shell data • We have to flatten our object out into three tables Wednesday, November 2, 11
28. Joins are great and all ... • Potentially organizationally messy • Structure of a single object is NOT immediately clear to someone glancing at the shell data • We have to flatten our object out into three tables • 7 separate inserts just to add “Programming in Scala” Wednesday, November 2, 11
29. Joins are great and all ... • Potentially organizationally messy • Structure of a single object is NOT immediately clear to someone glancing at the shell data • We have to flatten our object out into three tables • 7 separate inserts just to add “Programming in Scala” • Once we turn the relational data back into objects ... Wednesday, November 2, 11
30. Joins are great and all ... • Potentially organizationally messy • Structure of a single object is NOT immediately clear to someone glancing at the shell data • We have to flatten our object out into three tables • 7 separate inserts just to add “Programming in Scala” • Once we turn the relational data back into objects ... • We still need to convert it to data for our frontend Wednesday, November 2, 11
31. Joins are great and all ... • Potentially organizationally messy • Structure of a single object is NOT immediately clear to someone glancing at the shell data • We have to flatten our object out into three tables • 7 separate inserts just to add “Programming in Scala” • Once we turn the relational data back into objects ... • We still need to convert it to data for our frontend • I don’t know about you, but I have better things to do with my time. Wednesday, November 2, 11
32. The Same Data in MongoDB > db.books.find().forEach(printjson) { "_id" : ObjectId("4dfa6baa9c65dae09a4bbda3"), "title" : "The Demon-Haunted World: Science as a Candle in the Dark", "author" : [ { "first_name" : "Carl", "last_name" : "Sagan", "middle_name" : "Edward", "year_of_birth" : 1934 } ] } { "_id" : ObjectId("4dfa6baa9c65dae09a4bbda4"), "title" : "Cosmos", "author" : [ { "first_name" : "Carl", "last_name" : "Sagan", "middle_name" : "Edward", "year_of_birth" : 1934 } ] } Wednesday, November 2, 11
33. The Same Data in MongoDB (Part 2) { "_id" : ObjectId("4dfa6baa9c65dae09a4bbda5"), "title" : "Programming in Scala", "author" : [ { "first_name" : "Martin", "last_name" : "Odersky", "nationality" : "DE", "year_of_birth" : 1958 }, { "first_name" : "Lex", "last_name" : "Spoon" }, { "first_name" : "Bill", "last_name" : "Venners" } ] } Wednesday, November 2, 11
34. Access to the embedded objects is integral > db.books.find({"author.first_name": "Martin", "author.last_name": "Odersky"}) { "_id" : ObjectId("4dfa6baa9c65dae09a4bbda5"), "title" : "Programming in Scala", "author" : [ { "first_name" : "Martin", "last_name" : "Odersky", "nationality" : "DE", "year_of_birth" : 1958 }, { "first_name" : "Lex", "last_name" : "Spoon" }, { "first_name" : "Bill", "last_name" : "Venners" } ] } Wednesday, November 2, 11
35. As is manipulation of the embedded data > db.books.update({"author.first_name": "Bill", "author.last_name": "Venners"}, ... {$set: {"author.$.company": "Artima, Inc."}}) > db.books.update({"author.first_name": "Martin", "author.last_name": "Odersky"}, ... {$set: {"author.$.company": "Typesafe, Inc."}}) > db.books.findOne({"title": /Scala$/}) { "_id" : ObjectId("4dfa6baa9c65dae09a4bbda5"), "author" : [ { "company" : "Typesafe, Inc.", "first_name" : "Martin", "last_name" : "Odersky", "nationality" : "DE", "year_of_birth" : 1958 }, { "first_name" : "Lex", "last_name" : "Spoon" }, { "company" : "Artima, Inc.", "first_name" : "Bill", "last_name" : "Venners" } ], "title" : "Programming in Scala" } Wednesday, November 2, 11
36. NoSQL Really Means... non-relational, next-generation operational datastores and databases Wednesday, November 2, 11
37. NoSQL Really Means... non-relational, next-generation operational datastores and databases ... Let’s focus on the “non-relational” bit. Wednesday, November 2, 11
38. Horizontally Scalable Architectures no joins + no complex transactions Wednesday, November 2, 11
39. no joins + no complex transactions Wednesday, November 2, 11
40. no joins + no complex transactions Wednesday, November 2, 11
41. New Data Models no joins + no complex transactions Wednesday, November 2, 11
42. Best Use Cases Web Applications “Scaling Out” High Volume Traffic Caching Wednesday, November 2, 11
43. Less Suited For highly transactional applications problems which require SQL ad-hoc business intelligence Wednesday, November 2, 11
44. •MongoDB revolves around memory mapped files Memory Wednesday, November 2, 11
45. Operating System map files on the Filesystem to Virtual Memory Wednesday, November 2, 11
46. •(200 gigs of MongoDB files creates 200 gigs of virtual memory) Operating System map files on the Filesystem to Virtual Memory Wednesday, November 2, 11
47. •(200 gigs of MongoDB files creates 200 gigs of virtual memory) •OS controls what data in RAM Operating System map files on the Filesystem to Virtual Memory Wednesday, November 2, 11
48. •(200 gigs of MongoDB files creates 200 gigs of virtual memory) •OS controls what data in RAM •When a piece of data isn't found, a page fault occurs (Expensive + Locking!) Operating System map files on the Filesystem to Virtual Memory Wednesday, November 2, 11
49. •(200 gigs of MongoDB files creates 200 gigs of virtual memory) •OS controls what data in RAM •When a piece of data isn't found, a page fault occurs (Expensive + Locking!) •OS goes to disk to fetch the data Operating System map files on the Filesystem to Virtual Memory Wednesday, November 2, 11
50. •(200 gigs of MongoDB files creates 200 gigs of virtual memory) •OS controls what data in RAM •When a piece of data isn't found, a page fault occurs (Expensive + Locking!) •OS goes to disk to fetch the data •Compare this to the normal trick of sticking a poorly managed memcached cluster in front of MySQL Operating System map files on the Filesystem to Virtual Memory Wednesday, November 2, 11
51. _id if not specified drivers will add default: ObjectId("4bface1a2231316e04f3c434") timestamp machine id process id counter http://www.mongodb.org/display/DOCS/Object+IDs Wednesday, November 2, 11
52. BSON Encoding { _id: ObjectId(XXXXXXXXXXXX), hello: “world”} \x27\x00\x00\x00\x07_id\x00 X X X X X X X X X X X X X X \x02 h e l l o \x00\x06\x00 \x00\x00 w o r l d \x00\x00 http://bsonspec.org Wednesday, November 2, 11
53. bsonspec.org Wednesday, November 2, 11
54. Modeling Schemas in MongoDB • Think “Documents” not “Rows” • Documents are arranged in “collections” just as “rows” are organized in “tables” • Documents can be embedded in one another, or in Lists, etc. • Favor embedding over referencing • No datastore enforced foreign key relationships Wednesday, November 2, 11
55. MapReduce • MongoDB offers an implementation of MapReduce for calculating data aggregation • Functions are written in JavaScript • Support for “Incremental” Jobs to build on prior job outputs as data changes w/o rerunning whole dataset • Limitations of JavaScript engines reduce parallelism* Wednesday, November 2, 11
56. Geospatial Indexing “Where the hell am I?” • Search by (2D) Geospatial proximity with MongoDB • One GeoIndex per collection • Can index on an array or a subdocument • Searches against the index can treat the dataset as flat (map-like), Spherical (like a globe), and complex (box/ rectangle, circles, concave polygons and convex poylgons) Wednesday, November 2, 11
57. GridFS File Storage • Specification for storing large files in MongoDB; supported in all official drivers as reference implementation • Works around BSON document size limits by breaking files into chunks • Two collections: ‘fs.files’ for metadata, ‘fs.chunks’ for individual file pieces • Sharding: Individual file chunks don’t shard, but files themselves will • Experimental Modules for Lighttpd and Nginx • On JVM: get back java.io.File handles on GridFS read! Wednesday, November 2, 11
58. Scaling Wednesday, November 2, 11
59. Scaling • Operations/sec go up • Storage needs go up • Capacity • IOPs • Complexity goes up • Caching Wednesday, November 2, 11
60. • Optimization & Tuning • Schema & Index Design • O/S tuning • Hardware configuration • Vertical scaling • Hardware is expensive • Hard to scale in cloud How do you scale now? $$$ throughput Wednesday, November 2, 11
61. MongoDB Scaling - Single Node write read node_a1 Wednesday, November 2, 11
62. Read scaling - add Replicas write read node_b1 node_a1 Wednesday, November 2, 11
63. Read scaling - add Replicas write read node_c1 node_b1 node_a1 Wednesday, November 2, 11
64. Write scaling - Sharding write read shard1 node_c1 node_b1 node_a1 Wednesday, November 2, 11
65. Write scaling - add Shards write read shard1 node_c1 node_b1 node_a1 shard2 node_c2 node_b2 node_a2 Wednesday, November 2, 11
66. Write scaling - add Shards write read shard1 node_c1 node_b1 node_a1 shard2 node_c2 node_b2 node_a2 shard3 node_c3 node_b3 node_a3 Wednesday, November 2, 11
67. MongoDB and Scala Wednesday, November 2, 11
68. “ This s%#! doesn’t work the way I want it to” ‘mongo-scala-wrappers’ Is Born Wednesday, November 2, 11
69. “ This s%#! doesn’t work the way I want it to” ‘mongo-scala-wrappers’ Is Born • Learned MongoDB from Python Wednesday, November 2, 11
70. “ This s%#! doesn’t work the way I want it to” ‘mongo-scala-wrappers’ Is Born • Learned MongoDB from Python • Dynamic language with flexible syntax; Dynamic database with flexible schemas Wednesday, November 2, 11
71. “ This s%#! doesn’t work the way I want it to” ‘mongo-scala-wrappers’ Is Born • Learned MongoDB from Python • Dynamic language with flexible syntax; Dynamic database with flexible schemas • Tooling for MongoDB + Scala was limited or unsuited. Mostly focused on ODM. None of what I loved about Scala or MongoDB possible together. Wednesday, November 2, 11
72. “ This s%#! doesn’t work the way I want it to” ‘mongo-scala-wrappers’ Is Born • Learned MongoDB from Python • Dynamic language with flexible syntax; Dynamic database with flexible schemas • Tooling for MongoDB + Scala was limited or unsuited. Mostly focused on ODM. None of what I loved about Scala or MongoDB possible together. • Java Driver ... No Scala sugar or tricks Wednesday, November 2, 11
73. “ This s%#! doesn’t work the way I want it to” ‘mongo-scala-wrappers’ Is Born • Learned MongoDB from Python • Dynamic language with flexible syntax; Dynamic database with flexible schemas • Tooling for MongoDB + Scala was limited or unsuited. Mostly focused on ODM. None of what I loved about Scala or MongoDB possible together. • Java Driver ... No Scala sugar or tricks • scamongo (pre-lift): ODM (ORMey) or JSON tools Wednesday, November 2, 11
74. “ This s%#! doesn’t work the way I want it to” ‘mongo-scala-wrappers’ Is Born • Learned MongoDB from Python • Dynamic language with flexible syntax; Dynamic database with flexible schemas • Tooling for MongoDB + Scala was limited or unsuited. Mostly focused on ODM. None of what I loved about Scala or MongoDB possible together. • Java Driver ... No Scala sugar or tricks • scamongo (pre-lift): ODM (ORMey) or JSON tools • mongo-scala-driver: A little syntactic sugar but mostly ODM; didn’t “get” it Wednesday, November 2, 11
75. MongoDB from Python 1 doc = { 2 "name": { 3 "first": "Brendan", 4 "last": "McAdams" 5 }, 6 "email": "brendan@10gen.com", 7 "twitter": "@rit", 8 "age": 32, 9 "interests": ["scala", "python", "akka", "mongodb"] 10 } 11 12 age = doc['age'] 13 14 type(age) # <type 'int'> 15 16 doc['interests'][1] # 'python' 17 type(doc['interests']) # <type 'list'> Wednesday, November 2, 11
76. MongoDB from Java (or Scala, pre-Casbah) 1 val b = BasicDBObjectBuilder.start() 2 3 b.add("name", new BasicDBObject("first", "Brendan").append("last", "McAdams")) 4 b.add("email", "brendan@10gen.com") 5 b.add("twitter", "@rit") 6 b.add("age", 32) 7 8 val interests = new BasicDBList() 9 interests.add("scala") 10 interests.add("python") 11 interests.add("akka") 12 interests.add("mongodb") 13 14 b.add("interests", interests) 15 16 val doc = b.get() 17 18 val age = doc("age") // AnyRef = 32 19 20 doc("interests")(1) 21 /* error: AnyRef does not take parameters 22 doc("interests")(1) 23 */ 24 25 doc("interests").asInstanceOf[BasicDBList](1) // java.lang.Object = python 26 Wednesday, November 2, 11
77. Type Safety and Compilation Shouldn’t Necessitate Syntactic Suicide Wednesday, November 2, 11
78. Type Safety and Compilation Shouldn’t Necessitate Syntactic Suicide • There’s absolutely nothing wrong with that Syntax... For Java. Wednesday, November 2, 11
79. Type Safety and Compilation Shouldn’t Necessitate Syntactic Suicide • There’s absolutely nothing wrong with that Syntax... For Java. • Scala is expressive, fluid and beautiful; so is (IMHO) MongoDB. Wednesday, November 2, 11
80. Type Safety and Compilation Shouldn’t Necessitate Syntactic Suicide • There’s absolutely nothing wrong with that Syntax... For Java. • Scala is expressive, fluid and beautiful; so is (IMHO) MongoDB. • My goal: Teach Scala to be as close to Python / Mongo Shell as possible Wednesday, November 2, 11
81. Type Safety and Compilation Shouldn’t Necessitate Syntactic Suicide • There’s absolutely nothing wrong with that Syntax... For Java. • Scala is expressive, fluid and beautiful; so is (IMHO) MongoDB. • My goal: Teach Scala to be as close to Python / Mongo Shell as possible • Self Imposed Limitation: Don’t reinvent the wheel. Java Driver’s Network Layers, BSON Encoding, etc work great. Wednesday, November 2, 11
82. Type Safety and Compilation Shouldn’t Necessitate Syntactic Suicide • There’s absolutely nothing wrong with that Syntax... For Java. • Scala is expressive, fluid and beautiful; so is (IMHO) MongoDB. • My goal: Teach Scala to be as close to Python / Mongo Shell as possible • Self Imposed Limitation: Don’t reinvent the wheel. Java Driver’s Network Layers, BSON Encoding, etc work great. • Just add Syntactic Sugar! Wednesday, November 2, 11
83. Today ... 1 val doc = MongoDBObject( 2 "name" -> MongoDBObject("first" -> "Brendan", "last" -> "McAdams"), 3 "email" -> "brendan@10gen.com", 4 "twitter" -> "@rit", 5 "age" -> 32, 6 "interests"-> Seq("scala", "python", "akka", "mongodb") 7 ) 8 // Full support also for Scala 2.8 Collections' Factory / Builders 9 10 val age = doc.getAs[Int]("age") // Option[Int] = Some(32) 11 12 val interests = doc.as[Seq[_]]("interests") // Seq[java.lang.String] = List(scala, python, akka, mongodb) 13 14 interests(2) // akka 15 16 // Experimental Dynamic support in 2.9 lets you do doc.age.typed[Int] Wednesday, November 2, 11
84. It’s All About Implicits • Casbah’s core tenet is to reuse what works, and improve (not replace) what doesn’t... • Casbah provides a series of wrapper classes (and in some cases, companion objects) which proxy the “core” Java driver classes to provide scala functionality. • Generally provide “Scala-esque” wrappers to the MongoDB Java objects whereever possible. • These make sure to make iterable things Iterable, Cursors implement Iterator, DBObjects act like Scala Maps, etc. Wednesday, November 2, 11
85. • This is a different test collection I sometimes use for development & integration • There are 333 books, with rich data such as tags, pricing, etc. screen scraped from Amazon.com earlier this year • Let’s use this to better evaluate how we might use MongoDB + Scala A Somewhat Richer “books” Collection Wednesday, November 2, 11
86. A Somewhat Richer “books” Collection > db.books.findOne({"title": "Programming in Scala"}) { "_id" : ObjectId("4d59b68dcad4987053000161"), "author" : [ "Martin Odersky", "Lex Spoon", "Bill Venners" ], "isbn" : "0981531601", "price" : { "currency" : "USD", "discount" : 31.17, "msrp" : 49.99 }, "publicationYear" : 2008, "publisher" : "Artima, Inc.", "tags" : [ "scala", "functional programming", "java", "object oriented", "programming languages", "concurrent programming", "programming", /* ... */ "jvm", "object-oriented design", "static typing" ], "title" : "Programming in Scala" } Wednesday, November 2, 11
87. Accessing “books” from Casbah /** Import Casbah and load all of its implicit conversions, etc */ import com.mongodb.casbah.Imports._ val conn = MongoConnection() // could also say MongoConnection("localhost", 27017) val db: MongoDB = conn("bookstore") val mongo: MongoCollection = db("books") /** * Accessing all of the data is very easy in Casbah */ for (doc <- mongo) { println("Document: %s".format(doc)) } /** * Also doable more idiomatically with 'foreach' */ mongo.foreach{ doc => println("Document: %s".format(doc)) } Wednesday, November 2, 11
88. toString outputs valid JSON Document: { "_id" : { "$oid" : "4d59b6aecad4987053000173"} , "author" : [ "Benjamin C. Pierce"] , "isbn" : "0262660717" , "price" : { "currency" : "USD" , "discount" : 25.8 , "msrp" : 27} , "publicationYear" : 1991 , "publisher" : "The MIT Press" , "tags" : [ "category theory" , "mathematics" , "computer science" , "discrete mathematics" , "computer mathematics" , "logic" , "types"] , "title" : "Basic Category Theory For Computer Scientists"} Document: { "_id" : { "$oid" : "4d59b6b6cad4987053000174"} , "author" : [ "Benjamin C. Pierce"] , "isbn" : "9780262162098" , "price" : { "currency" : "USD" , "discount" : 53.64 , "msrp" : 76} , "publicationYear" : 2002 , "publisher" : "MIT Press" , "tags" : [ "type theory" , "computer science" , "functional programming" , "programming languages" , "types" , "cs" , "languages" , "programming" , "theory" , "compilers" , "lambda calculus" , "operational semantic" , "programming language theory" , "software development" , "static typing" , "tapl"] , "title" : "Types and Programming Languages"} Wednesday, November 2, 11
89. Idiomatic Queries, Anyone? /** * Querying in MongoDB can be done via special '$' operators * Which get a bit... messy in compiled languages */ val docQ = MongoDBObject( "publicationYear" -> MongoDBObject("$ne" -> 2008), "price.discount" -> MongoDBObject("$gt" -> 30, "$lt" -> 40) ) /** * Alternately, Casbah offers a Query DSL * which *explicitly* uses MongoDB syntax but in a more * natural Scala form. */ val dslQ = { "publicationYear" $ne 2008 } ++ { "price.discount" $gt 30 $lt 40 } /** * Both statements are identical! scala> dslQ == docQ res9: Boolean = true */ mongo.find(dslQ) Wednesday, November 2, 11
90. • It’s not always the most productive thing to work entirely in Maps (or in this case, techically ‘MongoDBObjects’) • We need something ‘richer’ for our domain code • Before we look at the obvious solution (ORMs), let’s look at how we might do this in regular old Scala + Casbah Often we need to map domain code to data, however Wednesday, November 2, 11
91. import com.mongodb.casbah.Imports._ case class Book(id: ObjectId, author: Seq[Author], isbn: String, price: Price, publicationYear: Int, tags: Seq[String], title: String, publisher: String, edition: Option[String]) { def toDBObject = MongoDBObject( "author" -> author.map { a => a.name }, "_id" -> id, "isbn" -> isbn, "price" -> price.toDBObject, "publicationYear" -> publicationYear, "tags" -> tags, "title" -> title, "publisher" -> publisher, "edition" -> edition ) } case class Author(name: String) case class Price(currency: String, discount: Double, msrp: Double) { def toDBObject = MongoDBObject( "currency" -> currency, "discount" -> discount, "msrp" -> msrp ) } Wednesday, November 2, 11
92. object Books extends App { val mongo = MongoConnection()("bookstore")("books") findAll().foreach(b => println("<Book> " + b)) val tim = Author("Timothy Perrett") val authors = Seq(tim) val price = Price("USD", 39.99, 39.99) val tags = Seq("functional programming", "scala", "web development", "lift", "#legendofklang") val liftInAction = Book(new ObjectId, authors, "9781935182801", price, 2011, tags, "Lift in Action: The Simply Functional Web Framework for Scala", "Manning Publications Co.", Some("First")) mongo.insert(liftInAction.toDBObject) def findAll() = for ( book <- mongo.find() ) yield newBook(book) def newBook(doc: MongoDBObject) = Book( doc.as[ObjectId]("_id"), doc.as[BasicDBList]("author").map(a => Author(a.asInstanceOf[String])).toSeq, doc.as[String]("isbn"), newPrice(doc.as[DBObject]("price")), doc.as[Number]("publicationYear").intValue, doc.as[BasicDBList]("tags").map(_.toString).toSeq, doc.as[String]("title"), doc.as[String]("publisher"), doc.getAs[String]("edition") ) Wednesday, November 2, 11
93. def newPrice(doc: MongoDBObject) = Price( doc.as[String]("currency"), doc.as[Number]("discount").doubleValue, doc.as[Number]("msrp").doubleValue ) } Wednesday, November 2, 11
94. Object Document Mapping Wednesday, November 2, 11
95. A New Paradigm for Mapping Objects <-> Documents Wednesday, November 2, 11
96. A New Paradigm for Mapping Objects <-> Documents • While the ORM Pattern can be a disaster, well designed Documents map well to a typical object hierarchy Wednesday, November 2, 11
97. A New Paradigm for Mapping Objects <-> Documents • While the ORM Pattern can be a disaster, well designed Documents map well to a typical object hierarchy • The world of ODMs for MongoDB has evolved in many languages, with fantastic tools in Scala, Java, Python and Ruby Wednesday, November 2, 11
98. A New Paradigm for Mapping Objects <-> Documents • While the ORM Pattern can be a disaster, well designed Documents map well to a typical object hierarchy • The world of ODMs for MongoDB has evolved in many languages, with fantastic tools in Scala, Java, Python and Ruby • Typically “relationship” fields can be defined to be either “embedded” or “referenced” Wednesday, November 2, 11
99. ODM Systems for the JVM < Scala > Wednesday, November 2, 11
100. ODM Systems for the JVM < Scala > • Two Major ODMs in the Scala World Wednesday, November 2, 11
101. ODM Systems for the JVM < Scala > • Two Major ODMs in the Scala World • Lift-MongoDB-Record Wednesday, November 2, 11
102. ODM Systems for the JVM < Scala > • Two Major ODMs in the Scala World • Lift-MongoDB-Record • Based on the Record pattern Wednesday, November 2, 11
103. ODM Systems for the JVM < Scala > • Two Major ODMs in the Scala World • Lift-MongoDB-Record • Based on the Record pattern • Requires entirely custom objects following Record paradigm Wednesday, November 2, 11
104. ODM Systems for the JVM < Scala > • Two Major ODMs in the Scala World • Lift-MongoDB-Record • Based on the Record pattern • Requires entirely custom objects following Record paradigm • Strongly coupled to MongoDB but still bound by Record Wednesday, November 2, 11
105. ODM Systems for the JVM < Scala > • Two Major ODMs in the Scala World • Lift-MongoDB-Record • Based on the Record pattern • Requires entirely custom objects following Record paradigm • Strongly coupled to MongoDB but still bound by Record • Bonus: Absolutely incredible “Rogue” DSL from Foursquare for taking Lift-MongoDB-Record further Wednesday, November 2, 11
106. ODM Systems for the JVM < Scala > • Two Major ODMs in the Scala World • Lift-MongoDB-Record • Based on the Record pattern • Requires entirely custom objects following Record paradigm • Strongly coupled to MongoDB but still bound by Record • Bonus: Absolutely incredible “Rogue” DSL from Foursquare for taking Lift-MongoDB-Record further • Salat Wednesday, November 2, 11
107. ODM Systems for the JVM < Scala > • Two Major ODMs in the Scala World • Lift-MongoDB-Record • Based on the Record pattern • Requires entirely custom objects following Record paradigm • Strongly coupled to MongoDB but still bound by Record • Bonus: Absolutely incredible “Rogue” DSL from Foursquare for taking Lift-MongoDB-Record further • Salat • Built by same team who helped start Casbah (scala driver) Wednesday, November 2, 11
108. ODM Systems for the JVM < Scala > • Two Major ODMs in the Scala World • Lift-MongoDB-Record • Based on the Record pattern • Requires entirely custom objects following Record paradigm • Strongly coupled to MongoDB but still bound by Record • Bonus: Absolutely incredible “Rogue” DSL from Foursquare for taking Lift-MongoDB-Record further • Salat • Built by same team who helped start Casbah (scala driver) • Annotation driven, built from ground up to apply onto existing business objects cleanly Wednesday, November 2, 11
109. ODM Systems for the JVM < Scala > • Two Major ODMs in the Scala World • Lift-MongoDB-Record • Based on the Record pattern • Requires entirely custom objects following Record paradigm • Strongly coupled to MongoDB but still bound by Record • Bonus: Absolutely incredible “Rogue” DSL from Foursquare for taking Lift-MongoDB-Record further • Salat • Built by same team who helped start Casbah (scala driver) • Annotation driven, built from ground up to apply onto existing business objects cleanly • Strongly coupled to MongoDB Wednesday, November 2, 11
110. Object Mapping with Lift import net.liftweb.mongodb.record._ import net.liftweb.mongodb.record.field._ import net.liftweb.record.field._ class Book private() extends BsonRecord[Book] with ObjectIdPk[Book] { def meta = Book object author extends MongoListField[Book, String](this) object isbn extends StringField(this, 64) object price extends BsonRecordField(this, Price) object publicationYear extends IntField(this) object tags extends MongoListField[Book, String](this) object title extends StringField(this, 255) object publisher extends StringField(this, 128) object edition extends StringField(this, 32) { override def optional_? = true } } object Book extends Book with MongoMetaRecord[Book] class Price private() extends BsonRecord[Price] { def meta = Price object currency extends StringField(this, 3) object discount extends DoubleField(this) object msrp extends DoubleField(this) } Wednesday, November 2, 11
111. Object Mapping with Lift val price = Price.createRecord.currency("USD").discount(39.99).msrp(39.99) val liftInAction = Book.createRecord.author(Seq("Timothy Perrett")). isbn("9781935182801"). price(price). publicationYear(2011). tags(Seq("functional programming", "scala", "web development", "lift", "#legendofklang")). title("Lift in Action: The Simply Functional Web Framework for Scala"). publisher("Manning Publications Co."). edition("First").save Wednesday, November 2, 11
112. Object Mapping with Salat /** Using our existing "Books" case classes, Salat can simply * Serialize them to MongoDB with it's "grater system" */ val tim = Author("Timothy Perrett") val authors = Seq(tim) val price = Price("USD", 39.99, 39.99) val tags = Seq("functional programming", "scala", "web development", "lift", "#legendofklang") val liftInAction = Book(new ObjectId, authors, "9781935182801", price, 2011, tags, "Lift in Action: The Simply Functional Web Framework for Scala", "Manning Publications Co.", Some("First")) /** * Instead of our custom "toDBObject" method, * the Salat Grater uses runtime Scala type reflection to * generate a MongoDB Object. */ val dbo = grater[Book].asDBObject(liftInAction) mongo.save(dbo) Wednesday, November 2, 11
113. Summary Wednesday, November 2, 11
114. Summary • New Datastores should mean new Paradigms Wednesday, November 2, 11
115. Summary • New Datastores should mean new Paradigms • No more should your application code be slave to the whims of your Database (or worse, your DBA) Wednesday, November 2, 11
116. Summary • New Datastores should mean new Paradigms • No more should your application code be slave to the whims of your Database (or worse, your DBA) • Regardless of which NoSQL solution you choose (Why not Zoidberg MongoDB?) Wednesday, November 2, 11
117. Summary Wednesday, November 2, 11
118. Summary • Brendan’s Laws of Happy Coding Wednesday, November 2, 11
119. Summary • Brendan’s Laws of Happy Coding • 1) The datastore is a servant to the application, not vice versa Wednesday, November 2, 11
120. Summary • Brendan’s Laws of Happy Coding • 1) The datastore is a servant to the application, not vice versa • 2) A usable datastore must offer out of the box solutions for simple, sane and proven Horizontal Scaling (No Frankensharding, please!)* Wednesday, November 2, 11
121. Summary • Brendan’s Laws of Happy Coding • 1) The datastore is a servant to the application, not vice versa • 2) A usable datastore must offer out of the box solutions for simple, sane and proven Horizontal Scaling (No Frankensharding, please!)* • 3) Man cannot live on whisky alone; sometimes he needs beer, too. Wednesday, November 2, 11
122. Summary Wednesday, November 2, 11
123. Summary Wednesday, November 2, 11
124. Summary Wednesday, November 2, 11
125. Summary • Anyone who tells you otherwise is a DBA, an Oracle Salesman or an Escaped Mental Patient Wednesday, November 2, 11
126. Summary • Anyone who tells you otherwise is a DBA, an Oracle Salesman or an Escaped Mental Patient • Sometimes, he’s all three ... Wednesday, November 2, 11
127. Summary • Anyone who tells you otherwise is a DBA, an Oracle Salesman or an Escaped Mental Patient • Sometimes, he’s all three ... • DISCLAIMER: This is not meant to disparage Oracle’s hiring practices, or the good name of Escaped Mental Patients everywhere Wednesday, November 2, 11
128. @mongodb conferences,  appearances,  and  meetups http://www.10gen.com/events http://bit.ly/mongoA   Facebook                    |                  Twitter                  |                  LinkedIn http://linkd.in/joinmongo download at mongodb.org MongoDB Ops & Admin Training @ SkillsMatter November 24 We’re Hiring ! brendan@10gen.com (twitter: @rit) Wednesday, November 2, 11