Rivieria Scala and Clojure - MongoDB talk

Slide 1

Slide 1 text

Riviera - Scala Clojure Ross Lawley Software Engineer, 10gen @RossC0

Slide 2

Slide 2 text

10gen and MongoDB

Slide 3

Slide 3 text

3 10gen Overview 10gen is the company behind MongoDB – the leading NoSQL database

Slide 4

Slide 4 text

4 10gen Overview Offices in New York, Palo Alto, Washington DC, London, Dublin, Barcelona and Sydney

Slide 5

Slide 5 text

MongoDB Overview

Slide 6

Slide 6 text

Relational Database Challenges Data Types • Unstructured data • Semi-structured data • Polymorphic data Volume of Data • Petabytes of data • Trillions of records • Tens of millions of queries per second Agile Development • Iterative • Short development cycles • New workloads New Architectures • Horizontal scaling • Commodity servers • Cloud computing

Slide 7

Slide 7 text

Database Landscape Depth of Functionality Scalability & Performance Memcached MongoDB RDBMS

Slide 8

Slide 8 text

MongoDB is a ___________ database • Document-oriented • Open-source • High performance and horizontally scalable • Full featured

Slide 9

Slide 9 text

Document-Oriented Database • Not .PDF & .DOC files • A document is essentially an associative array – JSON object, PHP Array, Python Dictionary, etc. • BSON – www.bsonspec.org

Slide 10

Slide 10 text

Open-Source • MongoDB is an open-source project • On GitHub • Database licensed under the AGPL • Drivers licensed under Apache

Slide 11

Slide 11 text

High Performance and Horizontally Scalable • High performance – Written in C++ – Data serialised as BSON (fast parsing) – Full support for primary & secondary indexes • Horizontally scalable – Auto-sharding – Scale across • Commodity hardware • Cloud compute • Hybrid

Slide 12

Slide 12 text

Full Featured • Rich Ad Hoc queries • Real time aggregation • Geospatial features • Native bindings for most programming languages

Slide 13

Slide 13 text

MongoDB is a Single-Master System • All writes are to a primary (master) • Failure of the primary is detected, and a new one is elected • Application writes get an error if there is no quorum to elect a new master – Reads can continue

Slide 14

Slide 14 text

MongoDB Storage Management • Data is kept in memory-mapped files • Files are allocated as needed • Indexes (B*-trees) point to documents using geographical addresses

Slide 15

Slide 15 text

Relational Schema User ·Name ·Email address Category ·Name ·URL Comment ·Comment ·Date ·Author Article ·Name ·Slug ·Publish date ·Text Tag ·Name ·URL

Slide 16

Slide 16 text

MongoDB Schema User ·Name ·Email address Article ·Name ·Slug ·Publish date ·Text ·Author Comment[] ·Comment ·Date ·Author Tag[] ·Value Category[] ·Value

Slide 17

Slide 17 text

Reliability – Replica Sets Secondary Secondary Primary Client Application Driver Write Read Read

Slide 18

Slide 18 text

Scalability – Auto-sharding Node 1 Secondary Config Server Node 1 Secondary Config Server Node 1 Secondary Config Server Shard Shard Shard Mongos App Server Mongos App Server Mongos App Server

Slide 19

Slide 19 text

Terminology RDBMS Mongo Table, View Collection Row(s) Document Index Index Join Linking/Embedding Partition Shard Partition Key Shard Key

Slide 20

Slide 20 text

Working with MongoDB

Slide 21

Slide 21 text

user = { username: "ross", first_name: "Ross", last_name: "Lawley"} > db.users.insert(user) Create (Insert)

Slide 22

Slide 22 text

> db.users.findOne() { "_id" : ObjectId("50ed3c5cab4ef39dc735664b"), "username" : "ross", "first_name" : "Ross", "last_name" : "Lawley" } Read (Query)

Slide 23

Slide 23 text

_id • _id is the primary key in MongoDB • Automatically indexed • Automatically created as an ObjectId if not provided • Any unique immutable value could be used

Slide 24

Slide 24 text

• ObjectId is a special 12 byte value • Guaranteed to be unique across your cluster ObjectId("50ed3c5cab4ef39dc735664b") |-------------||---------||-----||----------| ts mac pid inc ObjectId

Slide 25

Slide 25 text

// find users with any tags > db.users.find( {tags: {$exists: true }} ) // find users matching a regular expression > db.users.find( {username: /^ro*/i } ) // count posts by author > db.users.find( {username: "Ross"} ).count() • Conditional Operators – $all, $exists, $mod, $ne, $in, $nin, $nor, $or, $size, $type – $lt, $lte, $gt, $gte Query Operators

Slide 26

Slide 26 text

> tags = ["superuser", "db_admin"] > address = { street: "Scrutton Street", city: "London" } > db.users.update({}, {"$pushAll": {"tags": tags}, "$set": {"address": address}, "$inc": {"tag_count": 2}}) Update

Slide 27

Slide 27 text

> db.users.findOne() { "_id" : ObjectId("50ed3c5cab4ef39dc735664b"), "address" : { "street" : "Zetland House", "city" : "London" }, "first_name" : "Ross", "last_name" : "Lawley", "tag_count" : 2, "tags" : [ "superuser", "db_admin" ], "username" : "ross" } Read (Query)

Slide 28

Slide 28 text

• Scalar – $set, $unset, $inc, • Array – $push, $pushAll, $pull, $pullAll, $addToSet Atomic operators

Slide 29

Slide 29 text

// 1 means ascending, -1 means descending > db.users.ensureIndex({username: 1}) > db.users.find({username: "ross"}).explain() // Multi-key indexes > db.users.ensureIndex({tags: 1}) // index nested field > db.users.ensureIndex({"address.city": 1}) // Compound indexes > db.users.ensureIndex({ "username": 1, "address.city": 1 }) Secondary Indexes

Slide 30

Slide 30 text

MongoDB Drivers

Slide 31

Slide 31 text

Ofﬁcial 10Gen Drivers

Slide 32

Slide 32 text

MongoDB drivers • Official Support for 13 languages • Community drivers for tons more – Clojure, R, lua etc. • Drivers connect to mongo servers • Drivers translate BSON into native types • mongo shell is not a driver, but works like one in some ways • Installed using typical means (npm, pecl, gem, pip)

Slide 33

Slide 33 text

Scala

Slide 34

Slide 34 text

• Wraps the Java driver • Provide idiomatic Scala interface to MongoDB Scala

Slide 35

Slide 35 text

Scala case class Book(id: ObjectId, author: String, isbn: String, price: Price, year: Int, tags: Seq[String], title: String, publisher: String, edition: Option[String]) { def toDBObject = MongoDBObject( "_id" -> id, "author" -> author, "isbn" -> isbn, "price" -> price.toDBObject, "publicationYear" -> year, "tags" -> tags, "title" -> title, "publisher" -> publisher, "edition" -> edition ) }

Slide 36

Slide 36 text

Scala // Connect to default - localhost, 27017 scala> val mongoClient = MongoClient() mongoClient: com.mongodb.casbah.MongoClient ... // Chainable api scala> val mongoColl = mongoClient("bookstore")("books") mongoColl: com.mongodb.casbah.MongoCollection ...

Slide 37

Slide 37 text

Scala scala> val builder = MongoDBObject.newBuilder scala> builder += "foo" -> "bar" scala> builder += "x" -> "y" scala> builder += ("pie" -> 3.14) scala> builder += ("spam" -> "eggs", "mmm" -> "bacon") builder.type = com.mongodb.casbah.commons.MongoDBObjectBuilder // Return a DBObject scala> val newObj = builder.result newObj: com.mongodb.casbah.commons.Imports.DBObject = { "foo" : "bar" , "x" : "y" , "pie" : 3.14 , "spam" : "eggs" , "mmm" : "bacon"}

Slide 38

Slide 38 text

Scala scala> val newObj = MongoDBObject("foo" -> "bar", | "x" -> "y", | "pie" -> 3.14, | "spam" -> "eggs") newObj: com.mongodb.casbah.commons.Imports.DBObject = { "foo" : "bar" , "x" : "y" , "pie" : 3.14 , "spam" : "eggs"}

Slide 39

Slide 39 text

Scala val mongo: MongoClient = MongoClient()("bookstore")("books") def findAll() = for ( book <- mongo.find() ) yield new Book(book) findAll().foreach(b => println(" " + b))

Slide 40

Slide 40 text

Scala val mongo: MongoClient = MongoClient()("bookstore")("books") val query: DBObject = ("price" $lt 40.00) ++ ("tag" -> "scala") mongo.find( query )

Slide 41

Slide 41 text

• Wraps the Java driver • Scala-esque way of interacting with mongo • Casbah 2.5.0 coming soon! Casbah

Slide 42

Slide 42 text

Salat

Slide 43

Slide 43 text

• Bi-directional Scala case class serialization library • Leverages MongoDB's DBObject (target format) • Depends on the latest releases of: -scalap -casbah-core -mongo-java-driver • https://github.com/novus/salat Salat

Slide 44

Slide 44 text

Salat - there and back case class Alpha(x: String) scala> val a = Alpha(x = "Hello world") a: com.novus.salat.test.model.Alpha = Alpha(Hello world) scala> val dbo = grater[Alpha].asDBObject(a) dbo: com.mongodb.casbah.Imports.DBObject = { "_typeHint" : "com.novus.salat.test.model.Alpha" , "x" : "Hello world"} scala> val a_* = grater[Alpha].asObject(dbo) a_*: com.novus.salat.test.model.Alpha = Alpha(Hello world) scala> a == a_* res0: Boolean = true

Slide 45

Slide 45 text

• A case class instance extends Scala's Product trait, which provides a product iterator over its elements. • Salat used pickled Scala signatures to turn case classes into indexed fields with associated type information. • These fields are then serialized or deserialized using the memoized indexed fields with type information. How does that work?

Slide 46

Slide 46 text

• Case classes • Case classes typed to a trait or an abstract superclass (requires @Salat) • Inside a case class constructor • Immutable collections: lists, seqs, maps whose key is a String • Options • Any type handled by BSON encoding hooks Salat - Supports

Slide 47

Slide 47 text

• classes • case classes nested inside an enclosing class or trait (Cake pattern - coming soon) • collection support needs to be improved - no Set, Array, mutable Map, etc Salat - Limitations

Slide 48

Slide 48 text

Simple way to get started and create your own DAO • insert and get back an Option with the id • findOne - get back an Option typed to your case class • find and get back a Mongo cursor typed to your class • iterate, limit, skip and sort • update with a query and a case class • save and remove case classes SalatDAO

Slide 49

Slide 49 text

SalatDAO - example import com.novus.salat._ import com.novus.salat.global._ case class Omega(_id: ObjectId = new ObjectId, z: String, y: Boolean) object OmegaDAO extends SalatDAO[Omega, ObjectId]( collection = MongoConnection()("salat-example")("omega"))

Slide 50

Slide 50 text

SalatDAO - Insert and ﬁnd scala> val o = Omega(z = "something", y = false) o: Omega = Omega(4dac7b3e75e1b63949139c91, something, false) scala> val _id = OmegaDAO.insert(o) _id: Option[ObjectId] = Some(4dac7b3e75e1b63949139c91) scala> val o_* = OmegaDAO.findOne( MongoDBObject("z" -> "something")) o_*: Option[Omega] = Some( Omega(4dac7b3e75e1b63949139c91, something, false))

Slide 51

Slide 51 text

• Simplifies serialization to and from mongo • salatDAO provides a quick CRUD interface • New version coming soon! Salat

Slide 52

Slide 52 text

Scala - community drivers

Slide 53

Slide 53 text

• ReactiveMongo - asynchronous / non blocking by the guys at zenexity! Stephane Godbillon • Hammersmith - Netty + Akka.IO interfaces, strongly functional and callback based Scala - community drivers

Slide 54

Slide 54 text

• ReplicaSet support • Authentication support • GridFS support (streaming capable) • Cursors (providing a stream of documents) • Bulk inserts • Database commands support • Indexing operations ReactiveMongo

Slide 55

Slide 55 text

ReactiveMongo // select only my documents val query = BSONDocument("firstName" -> BSONString("Ross")) // get a Cursor[BSONDocument] val cursor = collection.find(query) //Enumerate this cursor and print a readable version cursor.enumerate.apply(Iteratee.foreach { doc => println("found document: " + DefaultBSONIterator.pretty(doc.bsonIterator)) })

Slide 56

Slide 56 text

ReactiveMongo // select only my documents val query = BSONDocument("firstName" -> BSONString("Ross")) // get a Cursor[BSONDocument] val cursor = collection.find(query) // Create a list using the toList helper val futurelist = cursor.toList futurelist.onSuccess { case list => val names = list.map( _.getAs[BSONString]("lastName").get.value) println("got names: " + names) }

Slide 57

Slide 57 text

Clojure - community drivers

Slide 58

Slide 58 text

http://clojuremongodb.info/ Monger

Slide 59

Slide 59 text

Monger (ns my.service.server (:require [monger.core :as mg]) (:import [com.mongodb MongoOptions ServerAddress])) ;; localhost, default port (mg/connect!) ;; set default database using set-db (mg/set-db! (mg/get-db "monger-test"))

Slide 60

Slide 60 text

Monger - Insert (connect!) (set-db! (monger.core/get-db "monger-test")) ;; with explicit document id (recommended) (insert "documents" { :_id (ObjectId.) :first_name "John" :last_name "Lennon" }) ;; with a different write concern (insert "documents" { :_id (ObjectId.) :first_name "John" :last_name "Lennon" } WriteConcern/JOURNAL_SAFE)

Slide 61

Slide 61 text

Monger - Querying (connect!) (set-db! (monger.core/get-db "monger-test")) ;; Query returns com.mongodb.DBObject object (mc/find "documents" {:first_name "Ringo"})

Slide 62

Slide 62 text

Monger - Querying (connect!) (set-db! (monger.core/get-db "monger-test")) ;; Query returns com.mongodb.DBObject object (mc/find "documents" {:first_name "Ringo"}) ;; returns documents as Clojure maps (mc/find-maps "documents" {:first_name "Ringo"})

Slide 63

Slide 63 text

• MongoDB is an agile, scalable NoSQL database • If you're coding on the JVM you have many options for interacting with MongoDB • Multiple native drivers • Some great community drivers In Summary

Slide 64

Slide 64 text

Click to edit Master text styles More Information Resource Location MongoDB Downloads www.mongodb.org/downloads Free Online Training education.10gen.com Webinars and Events www.10gen.com/events White Papers www.10gen.com/white-papers Customer Case Studies www.10gen.com/customers Presentations www.10gen.com/presentations Documentation docs.mongodb.org Additional Info [email protected]

Slide 65

Slide 65 text

• http://www.mongodb.org/display/DOCS/Drivers • http://reactivemongo.org/ • http://clojuremongodb.info/ More Information