Scala Days 2011

My talk at Scala Days 2011 at Stanford

18 Months With Scala

The Creation and Evolution of Casbah

Transcript

1. - 18 Months With Scala - Building a Driver For MongoDB Casbah (MongoDB + Scala) http://github.com/mongodb/casbah Brendan McAdams brendan@10gen.com @rit Scala Days 2011 June 3, 2011
2. - 18 Months With Scala - Building a Driver For MongoDB •"class Insanity extends Object {}": Getting Started With Scala • "This s%#! doesn't work the way I want it to": Where Casbah Came From ... (mongo-scala-wrappers) • "Do You Know What I Am Saying?": Pimping Java, Syntactic Sugar and Internal DSLs • "Rock The Casbah": Silly Names and 1.0 Releases • “They Got Me A Freaking Pony”: New Job, New Problems • "Eating Your Own Dog Food Isn't The Same As Making It Palatable": 1.1 becomes 2.0, Real Users
3. “ class Insanity extends Object {} “ Getting Started With Scala • 2009 “Learning Year” • Deeper Python, started to “get” Lambdas, FP • Learned C#; loved the “better Java” with lots of Pythonic functional stuff • Data Processing tools; Disco, Hadoop, OpenCL / CUDA, R, etc • New database technologies (NoSQL); Cassandra, Redis, CouchDB, Riak, MongoDB. • October 2009 ... • Put together NY NoSQL Conference (100+ ppl) • Job Imploded • New Job, New to Scala • October 2010 ... • Joined 10gen • Fulltime MongoDB Developer, work on Hadoop integration, Casbah & general Scala support as significant portion of my job
4. “ class Insanity extends Object {} “ Getting Started With Scala • Big Problems, New Tools needed • For much of it, Java wasn’t the answer • Scala brilliant tool for solving problems • Had read Wampler / Payne, not written code • Impulse Control Problem or Good Gut Feeling? • Akka huge part ... #legendofklang • Custom formulas, DSLs and other tools • Began fiddling with MongoDB tools for interstitial caching layer
5. “ This s%#! doesn’t work the way I want it to” ‘mongo-scala-wrappers’ Is Born • Learned MongoDB from Python • Dynamic language with flexible syntax; Dynamic database with flexible schemas • Tooling for MongoDB + Scala was limited or unsuited. Mostly focused on ODM. None of what I loved about Scala or MongoDB possible together. • Java Driver ... No Scala sugar or tricks • scamongo (pre-lift): ODM (ORMey) or JSON tools • mongo-scala-driver: A little syntactic sugar but mostly ODM; didn’t “get” it
6. MongoDB from Python 1 doc = { 2 "name": { 3 "first": "Brendan", 4 "last": "McAdams" 5 }, 6 "email": "brendan@10gen.com", 7 "twitter": "@rit", 8 "age": 31, 9 "interests": ["scala", "python", "akka", "mongodb"] 10 } 11 12 age = doc['age'] 13 14 type(age) # <type 'int'> 15 16 doc['interests'][1] # 'python' 17 type(doc['interests']) # <type 'list'>
7. MongoDB from Java (or Scala, pre-Casbah) 1 val b = BasicDBObjectBuilder.start() 2 3 b.add("name", new BasicDBObject("first", "Brendan").append("last", "McAdams")) 4 b.add("email", "brendan@10gen.com") 5 b.add("twitter", "@rit") 6 b.add("age", 31) 7 8 val interests = new BasicDBList() 9 interests.add("scala") 10 interests.add("python") 11 interests.add("akka") 12 interests.add("mongodb") 13 14 b.add("interests", interests) 15 16 val doc = b.get() 17 18 val age = doc("age") // AnyRef = 31 19 20 doc("interests")(1) 21 /* error: AnyRef does not take parameters 22 doc("interests")(1) 23 */ 24 25 doc("interests").asInstanceOf[BasicDBList](1) // java.lang.Object = python 26
8. Type Safety and Compilation Shouldn’t Necessitate Syntactic Suicide • There’s absolutely nothing wrong with that Syntax... For Java. • Scala is expressive, fluid and beautiful; so is (IMHO) MongoDB. • My goal: Teach Scala to be as close to Python / Mongo Shell as possible • Self Imposed Limitation: Don’t reinvent the wheel. Java Driver’s Network Layers, BSON Encoding, etc work great. • Just add Syntactic Sugar!
9. Today ... 1 val doc = MongoDBObject( 2 "name" -> MongoDBObject("first" -> "Brendan", "last" -> "McAdams"), 3 "email" -> "brendan@10gen.com", 4 "twitter" -> "@rit", 5 "age" -> 31, 6 "interests"-> Seq("scala", "python", "akka", "mongodb") 7 ) 8 // Full support also for Scala 2.8 Collections' Factory / Builders 9 10 val age = doc.getAs[Int]("age") // Option[Int] = Some(31) 11 12 val interests = doc.as[Seq[_]]("interests") // Seq[java.lang.String] = List(scala, python, akka, mongodb) 13 14 interests(2) // akka 15 16 // Experimental Dynamic support in 2.9 lets you do doc.age.typed[Int]
10. Today ... "Chained core operators" should { "Function correctly" in { val ltGt = "foo" $gte 15 $lt 35.2 $ne 16 log.debug("LTGT: %s", ltGt) ltGt must beEqualTo(MongoDBObject("foo" -> MongoDBObject("$gte" -> 15, "$lt" -> 35.2, "$ne" -> 16))) } "Function correctly with deeper nesting e.g. $not" in { val ltGt = "foo" $not { _ $gte 15 $lt 35.2 $ne 16 } log.debug("LTGT: %s", ltGt) ltGt must beEqualTo(MongoDBObject("foo" -> MongoDBObject("$not" -> MongoDBObject("$gte" -> 15, "$lt" -> 35.2, "$ne" -> 16)))) } } "with Long" in { val neLong = "foo" $lte 10L neLong must haveEntry("foo.$lte" -> 10L) } "with Short" in { val neShort = "foo" $lte java.lang.Short.parseShort("10") neShort must haveEntry("foo.$lte" -> java.lang.Short.parseShort("10")) } "with JDKDate" in { val neJDKDate = "foo" $lte testDate neJDKDate must haveEntry("foo.$lte" -> testDate) } "with JodaDT" in { RegisterJodaTimeConversionHelpers() val neJodaDT = "foo" $lte new org.joda.time.DateTime(testDate.getTime) neJodaDT must haveEntry("foo.$lte" -> new org.joda.time.DateTime(testDate.getTime)) }
11. ... But it took ~18 months to get there • Feb. 12, 2010: Initial Open Source Release (0.1) No Tests. - Initial import Compiles, reflects the working code currently in Novus Trunk but does not have full documentation, or tests yet. NOT FOR PUBLIC CONSUMPTION - USE AT YOUR OWN RISK. * Release 0.1 - May or may not blow your system up... - Updated headers, scaladoc/javadoc documentation, etc. - Next step: Written docs with examples, test classes • July 17, 2010: Release 1.0. • New collaborator/contributor Max Afonov (@max4f) • January 03, 2010: Release 2.0. • Refactoring & Stupidity cleanups.
12. “ Do You Know What I Am Saying? “ Pimping Java, Syntactic Sugar, & Internal DSLs
13. It took awhile to get things right 1 val doc = ( 2 "name" -> ("first" -> "Brendan", "last" -> "McAdams"), 3 "email" -> "brendan@10gen.com", 4 "twitter" -> "@rit", 5 "age" -> 31, 6 "interests" -> ("scala", "python", "akka", "mongodb") 7 ) // Tuple conversion looked nice but BOY was it problematic 8 9 // Syntax for getAs, etc came much, much later 10 val age = doc("age") // AnyRef = 31 11 12 doc("interests")(1) 13 /* error: AnyRef does not take parameters 14 doc("interests")(1) 15 */ 16 17 doc("interests").asInstanceOf[BasicDBList](1) // java.lang.Object = python 18
14. So very many things wrong with this code 1 /** 2 * Hacky mildly absurd method for converting a <code>Product</code> (Example being any <code>Tuple</ code>) to 3 * a Mongo <code>DBObject</code> on the fly to minimize spaghetti code from long builds of Maps or DBObjects. 4 * 5 * Intended to facilitate fluid code but may be dangerous. 6 * _ * SNIP 17 */ 18 implicit def productToMongoDBObject(p: Product): DBObject = { 19 val builder = BasicDBObjectBuilder.start 20 val arityRange = 0.until(p.productArity) 21 //println("Converting Product P %s with an Arity range of %s to a MongoDB Object".format(p, arityRange)) 22 for (i <- arityRange) { 23 val x = p.productElement(i) 24 //println("\tI: %s X: %s".format(i, x)) 25 if (x.isInstanceOf[Tuple2[_,_]]) { 26 val t = x.asInstanceOf[Tuple2[String, Any]] 27 //println("\t\tT: %s".format(t)) 28 builder.add(t._1, t._2) 29 } else if (p.productArity == 2 && p.productElement(0).isInstanceOf[String]) { 30 // backup plan if it's a one entry tuple, the outer wrapper gets stripped 31 val t = p.asInstanceOf[Tuple2[String, Any]] 32 builder.add(t._1, t._2) 33 return builder.get 34 } else { 35 throw new IllegalArgumentException("Products to convert to DBObject must contain Tuple2's.") 36 } 37 } 38 builder.get 39 } 40
15. In a Strongly Typed Language Like Java, Square Pegs do not fit Into Round Holes
16. In Loosely/Dynamically Typed Languages (Perl, Python, Ruby, etc) Round Holes can be convinced to Accept Square Pegs
17. Scala gives us a lathe instead ... • Implicits, Type Classes, Abstract & Parameterized Types (Scala’s variant, esp. with Covariance/Contravariance annotation), Structural (aka “sort of a duck” typing) are incredible • For third party libraries though... • “Am I helping my users or hurting them?” • “Have I accounted for all the use cases?” • “Do I have any idea what the f$%k I’m doing?”
18. But the rule is not “Measure once, cut twice” ... (Shamelessly  swiped  from  https://jeffpeachey.wordpress.com/2010/07/19/measure-­‐twice-­‐cut-­‐once/) - Manifest fun can protect you from a lot of compile time stupidity (so can @tailrec!) but when you’re doing runtime serialization it may not be enough. - Type Classes let you create type safe (or quasi-type safe) methods but still let your users add on to them. Important in a serialization arch where users can define custom class ser/deser - Manifests vs. Type Classes
19. But the rule is not “Measure once, cut twice” ... ...then again, I’m no carpenter • Much of what I love about Scala are often compile time checks and don’t keep you from misunderstanding things, hurting your users or just plain screwing up. • Fun with Type Inference aka “Oops, I screwed the explicit annotators” • Know and understand the “fancy” features, but also know when to use them. • Know when not to use them, and when one choice is better than another. “The difference between a junior and a senior programmer is that the senior knows when not to write code.” - Manifest fun can protect you from a lot of compile time stupidity (so can @tailrec!) but when you’re doing runtime serialization it may not be enough. - Type Classes let you create type safe (or quasi-type safe) methods but still let your users add on to them. Important in a serialization arch where users can define custom class ser/deser - Manifests vs. Type Classes
20. Learning New Features (however hard/undocumented) always made me a better Scala developer • How do you have a quasi-type safe (compile time “valid types” enforcement) Query DSL in a language/engine where users can define serialization of arbitrary custom types? • aka This code sucks trait LessThanEqualOp extends QueryOperator { def $lte(target: String) = op("$lte", target) def $lte(target: java.util.Date) = op("$lte", target) def $lte(target: AnyVal) = op("$lte", target) def $lte(target: DBObject) = op("$lte", target) def $lte(target: Map[String, Any]) = op("$lte", target.asDBObject) } Question lead in to me admitting my code sucked.
21. Type Classes for Safe Configurability trait LessThanEqualOp extends QueryOperator { private val oper = "$lte" def $lte(target: String) = op(oper, target) def $lte(target: DBObject) = op(oper, target) def $lte(target: Array[_]) = op(oper, target.toList) def $lte(target: Tuple1[_]) = op(oper, target.productIterator.toList) def $lte(target: Tuple2[_, _]) = op(oper, target.productIterator.toList) def $lte(target: Tuple3[_, _, _]) = op(oper, target.productIterator.toList) def $lte(target: Tuple4[_, _, _, _]) = op(oper, target.productIterator.toList) def $lte(target: Tuple5[_, _, _, _, _]) = op(oper, target.productIterator.toList) def $lte(target: Tuple6[_, _, _, _, _, _]) = op(oper, target.productIterator.toList) def $lte(target: Tuple7[_, _, _, _, _, _, _]) = op(oper, target.productIterator.toList) def $lte(target: Tuple8[_, _, _, _, _, _, _, _]) = op(oper, target.productIterator.toList) def $lte(target: Tuple9[_, _, _, _, _, _, _, _, _]) = op(oper, target.productIterator.toList) def $lte(target: Tuple10[_, _, _, _, _, _, _, _, _, _]) = op(oper, target.productIterator.toList) def $lte(target: Tuple11[_, _, _, _, _, _, _, _, _, _, _]) = op(oper, target.productIterator.toList) def $lte(target: Tuple12[_, _, _, _, _, _, _, _, _, _, _, _]) = op(oper, target.productIterator.toList) def $lte(target: Tuple13[_, _, _, _, _, _, _, _, _, _, _, _, _]) = op(oper, target.productIterator.toList) def $lte(target: Tuple14[_, _, _, _, _, _, _, _, _, _, _, _, _, _]) = op(oper, target.productIterator.toList) def $lte(target: Tuple15[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _]) = op(oper, target.productIterator.toList) def $lte(target: Tuple16[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _]) = op(oper, target.productIterator.toList) def $lte(target: Tuple17[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _]) = op(oper, target.productIterator.toList) def $lte(target: Tuple18[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _]) = op(oper, target.productIterator.toList) def $lte(target: Tuple19[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _]) = op(oper, target.productIterator.toList) def $lte(target: Tuple20[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _]) = op(oper, target.productIterator.toList) def $lte(target: Tuple21[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _]) = op(oper, target.productIterator.toList) def $lte(target: Tuple22[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _]) = op(oper, target.productIterator.toList) def $lte(target: Iterable[_]) = op(oper, target.toList) def $lte[T: ValidDateOrNumericType](target: T) = op(oper, target) }
22. Type Classes for Safe Configurability /** * User configurable filters! */ trait ValidDateOrNumericTypeHolder extends ValidDateTypeHolder with ValidNumericTypeHolder { implicit object JDKDateDoNOk extends JDKDateOk with ValidDateOrNumericType [java.util.Date] implicit object JodaDateTimeDoNOk extends JDKDateOk with ValidDateOrNumericType [org.joda.time.DateTime] implicit object BigIntDoNOk extends BigIntOk with ValidDateOrNumericType [BigInt] implicit object IntDoNOk extends IntOk with ValidDateOrNumericType[Int] implicit object ShortDoNOk extends ShortOk with ValidDateOrNumericType[Short] implicit object ByteDoNOk extends ByteOk with ValidDateOrNumericType[Byte] implicit object LongDoNOk extends LongOk with ValidDateOrNumericType[Long] implicit object FloatDoNOk extends FloatOk with ValidDateOrNumericType[Float] implicit object BigDecimalDoNOk extends BigDecimalOk with ValidDateOrNumericType[BigDecimal] implicit object DoubleDoNOk extends DoubleOk with ValidDateOrNumericType [Double] }
23. Figuring out Manifests was hard ... but valuable to emulate type safety def $type[A: BSONType: Manifest] = if (manifest[A] <:< manifest[Double]) op(oper, BSON.NUMBER) else if (manifest[A] <:< manifest[String]) op(oper, BSON.STRING) else if (manifest[A] <:< manifest[BasicDBList] || manifest[A] <:< manifest[BasicBSONList]) op(oper, BSON.ARRAY) else if (manifest[A] <:< manifest[BSONObject] || manifest[A] <:< manifest[DBObject]) op(oper, BSON.OBJECT) else if (manifest[A] <:< manifest[ObjectId]) op(oper, BSON.OID) else if (manifest[A] <:< manifest[Boolean]) op(oper, BSON.BOOLEAN) else if (manifest[A] <:< manifest[java.sql.Timestamp]) op(oper, BSON.TIMESTAMP) else if (manifest[A] <:< manifest[java.util.Date] || manifest[A] <:< manifest[org.joda.time.DateTime]) op(oper, BSON.DATE) else if (manifest[A] <:< manifest[Option[Nothing]]) op(oper, BSON.NULL) else if (manifest[A] <:< manifest[Regex]) op(oper, BSON.REGEX) else if (manifest[A] <:< manifest[Symbol]) op(oper, BSON.SYMBOL) else if (manifest[A] <:< manifest[Int]) op(oper, BSON.NUMBER_INT) else if (manifest[A] <:< manifest[Long]) op(oper, BSON.NUMBER_LONG) else if (manifest[A].erasure.isArray && manifest[A] <:< manifest[Array[Byte]]) op(oper, BSON.BINARY) else throw new IllegalArgumentException("Invalid BSON Type '%s' for matching".format(manifest.erasure)) * Manifests worked *GREAT* For solving this weird dynamic typing problem
24. Figuring out Manifests was hard, but valuable to emulate type safety /** * I had used Type classes elsewhere, but when I posted the preceding * manifest code as an example of cool stuff to show @ ScalaDays, * Jon-Anders Teigen (@jteigen) sent me a gist with a better way. * Type Classes for this! */ def $type[A](implicit bsonType: BSONType[A]) = op(oper, bsonType.operator) /** * Thats now it for the $type support, it uses a few type class definitions as * well to match the BSON types. */ implicit object BSONDouble extends BSONType[Double](BSON.NUMBER) implicit object BSONString extends BSONType[String](BSON.STRING) implicit object BSONObject extends BSONType[BSONObject](BSON.OBJECT) implicit object DBObject extends BSONType[DBObject](BSON.OBJECT) implicit object DBList extends BSONType[BasicDBList](BSON.ARRAY) implicit object BSONDBList extends BSONType[BasicBSONList](BSON.ARRAY) implicit object BSONBinary extends BSONType[Array[Byte]](BSON.BINARY) implicit object BSONObjectId extends BSONType[ObjectId](BSON.OID) implicit object BSONBoolean extends BSONType[Boolean](BSON.BOOLEAN) implicit object BSONJDKDate extends BSONType[java.util.Date](BSON.DATE) implicit object BSONJodaDateTime extends BSONType[org.joda.time.DateTime](BSON.DATE) implicit object BSONNull extends BSONType[Option[Nothing]](BSON.NULL) implicit object BSONRegex extends BSONType[Regex](BSON.REGEX) implicit object BSONSymbol extends BSONType[Symbol](BSON.SYMBOL) implicit object BSON32BitInt extends BSONType[Int](BSON.NUMBER_INT) implicit object BSON64BitInt extends BSONType[Long](BSON.NUMBER_LONG) implicit object BSONSQLTimestamp extends BSONType[java.sql.Timestamp](BSON.TIMESTAMP)
25. “Rock The Casbah” Silly Names and 1.0 Releases • June 01, 2010: The death of “mongo-scala-wrappers” - Changed package to com.novus.casbah.mongodb - Rolled Scala version to 2.8.0.RC3 and SBT version to 0.7.4 - Updated dependency libraries as appropriate for 2.8rc3 - Cleaned up package declarations in code - Rolled module version to 1.0-SNAPSHOT as the next release goal is to be complete at a 1.0 • “Casbah” inspired mostly randomly from The Clash • "Rock The Casbah": Silly Names and 1.0 Releases • 1.1 Began on a mission of modularisation and functionality expansion • “casbah-mapper” borne unto git, never released mainstream and ultimately reimagined as “Salat” (Russian word салат for “salad”) • salat-avro (@rubbish) • @coda’s “Jerkson” project using some of the ScalaSig code utils from Salat Growth far beyond just being a wrapper Then again, a lot is in a name... Arguments against renaming later. “randomness of radio inspiration” Salat - The strength of an ecosystem (Being used for not just mongo ...
26. “They Got Me a Freaking Pony!” Pro  Tip:  Working  with  the  kind  of  people  who  will   *actually*  buy  you  a  pony  is  highly  recommended Not featured in the PDF version: Sparkly transition effect
27. “Eating Your Own Dog Food Isn’t The Same As Making It Palatable” • There’s a difference between “Fixing bugs in production” and “Shipping libraries to users” • Eating my own dog food was great, but in many ways it made me complacent • In many cases I initially only implemented MongoDB features I was using... • ... In others, only the way I was using them.
28. “Eating Your Own Dog Food Isn’t The Same As Making It Palatable” • 15 years as a developer taught me this: “Tests seem like a really good idea... I’m tired of fixing my broken crap in production” for (i <- 1 to ∞) println(“Tests. Matter.”)
29. “Eating Your Own Dog Food Isn’t The Same As Making It Palatable” • 15 years of reality tempered “nice to have” with “shutup and code”: “I wish I had time to actually write tests and learn to write good tests” <Boss> “Just put it in production and fix it later, we don’t have time to wait” • Let’s face it: This isn’t excuses but in many cases, reality. Ship code or flip burgers.
30. “Eating Your Own Dog Food Isn’t The Same As Making It Palatable” • If you plan to ship code to users, “eating your own dog food” is NEVER ENOUGH* • Take the time to learn how to write good tests and GOOD DATA • I am head over heels in love with the tools in Scala • ScalaTest (I don’t use anymore but still amazing) • Specs / Specs 2: Alien Technology for breaking my code • ScalaCheck - Haven’t learned it yet, but does fuzzing, etc • Differentiate between integration tests and unit tests • But *use* integration tests with “conditional skips”, and WRITE THEM. *  Assuming  of  course  you  care  about            code  quality  and/or  your  users Tons and tons of bugs found as I moved to specs2 , that had lurked under the surface for time immemorial
31. “Eating Your Own Dog Food Isn’t The Same As Making It Palatable” • Some of why I didn’t test Casbah as well early on is I couldn’t easily test the values as MongoDB saw them. • With moving to Specs2, it was much more strict and I was inspired to write custom matchers to do the job; provided for users too! (Tests all the way down...) • Tests are much cleaner and I feel more confident about them; able to achieve higher coverage • Higher coverage definitively relates to less bugs users find in their production apps Tons and tons of bugs found as I moved to specs2 , that had lurked under the surface for time immemorial
32. “Yak Shaving Becomes Yak Nairing” trait CasbahSpecification extends Specification with DBObjectMatchers with Logging { /** SNIP */ } trait DBObjectMatchers extends DBObjectBaseMatchers trait DBObjectBaseMatchers extends Logging { protected def someField(map: Expectable[Option[DBObject]], k: String) = if (k.indexOf('.') < 0) { map.value.getOrElse(MongoDBObject.empty).getAs[AnyRef](k) } else { map.value.getOrElse(MongoDBObject.empty).expand[AnyRef](k) } protected def field(map: Expectable[DBObject], k: String) = if (k.indexOf('.') < 0) { map.value.getAs[AnyRef](k) } else { map.value.expand[AnyRef](k) } protected def listField(map: Expectable[DBObject], k: String) = if (k.indexOf('.') < 0) { map.value.getAs[Seq[Any]](k) } else { map.value.expand[Seq[Any]](k) } def beDBObject: Matcher[AnyRef] = ((_: AnyRef).isInstanceOf[DBObject], " is a DBObject", " is not a DBObject") def haveSomeField(k: String) = new Matcher[Option[DBObject]] { def apply[S <: Option[DBObject]](map: Expectable[S]) = { result(someField(map, k).isDefined, map.description + " has the key " + k, map.description + " doesn't have the key " + k, map) } } /** matches if dbObject.contains(k) */ def haveField(k: String) = new Matcher[DBObject] { def apply[S <: DBObject](map: Expectable[S]) = { result(field(map, k).isDefined, map.description + " has the key " + k, map.description + " doesn't have the key " + k, map) } } /** matches if a Some(map) contains a pair (key, value) == (k, v) * Will expand out dot notation for matching. **/ def haveSomeEntry[V](p: (String, V)) = new Matcher[Option[DBObject]] { def apply[S <: Option[DBObject]](map: Expectable[S]) = { Tons and tons of bugs found as I moved to specs2 , that had lurked under the surface for time immemorial
33. “Yak Shaving Becomes Yak Nairing” /** matches if dbObject.contains(k) */ def haveField(k: String) = new Matcher[DBObject] { def apply[S <: DBObject](map: Expectable[S]) = { result(field(map, k).isDefined, map.description + " has the key " + k, map.description + " doesn't have the key " + k, map) } } /** matches if a Some(map) contains a pair (key, value) == (k, v) * Will expand out dot notation for matching. **/ def haveSomeEntry[V](p: (String, V)) = new Matcher[Option[DBObject]] { def apply[S <: Option[DBObject]](map: Expectable[S]) = { result(someField(map, p._1).exists(_ == p._2), // match only the value map.description + " has the pair " + p, map.description + " doesn't have the pair " + p, map) } } /** Special version of "HaveEntry" that expects a list and then uses * "hasSameElements" on it. */ def haveListEntry(k: String, l: => Traversable[Any]) = new Matcher[DBObject] { def apply[S <: DBObject](map: Expectable[S]) = { val objL = listField(map, k).getOrElse(Seq.empty[Any]).toSeq val _l = l.toSeq result(objL.sameElements(_l), // match only the value map.description + " has the pair " + k, map.description + " doesn't have the pair " + k, map) } } /** matches if map contains a pair (key, value) == (k, v) * Will expand out dot notation for matching. **/ def haveEntry[V](p: (String, V)) = new Matcher[DBObject] { def apply[S <: DBObject](map: Expectable[S]) = { result(field(map, p._1).exists(_.equals(p._2)), // match only the value map.description + " has the pair " + p, map.description + "[" + field(map, p._1) + "] doesn't have the pair " + p + "[" + p._2 + "]", map) } } /** matches if Some(map) contains all the specified pairs * can expand dot notation to match specific sub-keys */ def haveSomeEntries[V](pairs: (String, V)*) = new Matcher[Option[DBObject]] { def apply[S <: Option[DBObject]](map: Expectable[S]) = { result(pairs.forall(pair => someField(map, pair._1).exists(_ == pair._2) /* match only the value */ ), map.description + " has the pairs " + pairs.mkString(", "), map.description + " doesn't have the pairs " + pairs.mkString(", "), map) } } /** matches if map contains all the specified pairs Tons and tons of bugs found as I moved to specs2 , that had lurked under the surface for time immemorial
34. epilogue • Casbah lives on and will continue to evolve, but it also has a younger brother/cousin • “Hammersmith”, purely asynchronous, purely Scala and a distillation of ~2 years of MongoDB knowledge • Only Java is the BSON serialization; still no excuse for reinventing the wheel • Netty for now, but probably will end up as pure NIO • NOT (contrary to popular panic/confusion) a replacement for Casbah • Focused more on framework support than userspace • Will likely offer optional synchronous and asynchronous hammersmith module for casbah-core, with Java driver as casbah- core-classic • Working on sharing as much code as possible between Hammersmith & Casbah for MongoDBObject, etc. • Porting casbah-query to target Hammersmith (as well as Lift)
35. epilogue 1 def iterateSimpleCursor(conn: MongoConnection) = { 2 var x = 0 3 conn("bookstore").find("inventory")(Document.empty, Document.empty)((cursor: Cursor) => { 4 for (doc <- cursor) { 5 x += 1 6 } 7 }) 8 9 x must eventually (be_==(336)) 10 } 11 12 def iterateComplexCursor(conn: MongoConnection) = { 13 var x = 0 14 conn("bookstore").find("inventory")(Document.empty, Document.empty)((cursor: Cursor) => { 15 def next(op: Cursor.IterState): Cursor.IterCmd = op match { 16 case Cursor.Entry(doc) => { 17 x += 1 18 if (x < 100) Cursor.Next(next) else Cursor.Done 19 } 20 case Cursor.Empty => { 21 if (x < 100) Cursor.NextBatch(next) else Cursor.Done 22 } 23 case Cursor.EOF => { 24 Cursor.Done 25 } 26 } 27 Cursor.iterate(cursor)(next) 28 }) 29 30 x must eventually(5, 5.seconds) (be_==(100)) 31 } - Iteratees, No blocking calls, Callbacks and no Java wrappers or wackiness - Thanks to @prasinous, @jdegoes, @etorreborre who shared ideas, code and beat me about the head and neck at times for stupidity as I evolved this.
36. epilogue 33 def insertWithSafeImplicitWriteConcern(conn: MongoConnection) = { 34 val mongo = conn("testHammersmith")("test_insert") 35 implicit val safeWrite = WriteConcern.Safe 36 mongo.dropCollection()(success => { 37 log.info("Dropped collection... Success? " + success) 38 }) 39 var id: Option[AnyRef] = null 40 var ok: Option[Boolean] = None 41 42 val handler = RequestFutures.write((result: Either[Throwable, (Option[AnyRef], WriteResult)]) => { 43 result match { 44 case Right((oid, wr)) => { 45 ok = Some(true) 46 id = oid } 47 case Left(t) => { 48 ok = Some(false) 49 log.error(t, "Command Failed.") 50 } 51 } 52 }) 53 mongo.insert(Document("foo" -> "bar", "bar" -> "baz"))(handler) 54 ok must eventually { beSome(true) } 55 id must not (beNull.eventually) 56 // TODO - Implement 'count' 57 var doc: BSONDocument = null 58 mongo.findOne(Document("foo" -> "bar"))((_doc: BSONDocument) => { 59 doc = _doc 60 }) 61 doc must not (beNull.eventually) 62 doc must eventually (havePairs("foo" -> "bar", "bar" -> "baz")) 63 } 64 - Iteratees, No blocking calls, Callbacks and no Java wrappers or wackiness - Thanks to @prasinous, @jdegoes, @etorreborre who shared ideas, code and beat me about the head and neck at times for stupidity as I evolved this.
37. @mongodb International  Scala  Hackathon  (“Scalathon”)  Coming! July  16-­‐17,  2011  @  UPenn  (Philadelphia,  PA  USA) http://scalathon.org conferences,  appearances,  and  meetups http://www.10gen.com/events http://bit.ly/mongoH   Facebook                    |                  Twitter                  |                  LinkedIn http://linkd.in/joinmongo download at mongodb.org github.com/mongodb/casbah We’re Hiring ! ( http://10gen.jobscore.com/jobs/10gen/list ) brendan@10gen.com twitter: @rit slides: http://speakerdeck.com/u/bwmcadams/p/scala-days-2011 - Goal of Scalathon: to get Scala developers contributing to the language, its tools, and its libraries. * Attending: - Almost full but if response/interest continues, may be possible to open more seats soon!
38. Acknowledgements • Because this genuinely was never my effort alone, these people contributed patches, suggestions, bugs, ideas or were just early users... • Michael Merwitz (@michaelmerwitz) who suggested/encouraged we “give MongoDB a try” (Python/IronPython/C# project, pre-Novus/Casbah) • Gregg Carrier (@greggcarrier) • Gerolf Seitz (@gseitz) • Phil Wills (@philwills) • Max Afonov (@max4f) • Marcello Bastéa-Forte (@marcello3d) • Rose Toomey (@prasinous) (Her contributions are probably largest in many ways) • Novus Partners, esp. Basil (CEO) for incubating and cultivating Casbah and allowing it (and me) to grow beyond his company. • 10gen ... Eliot, Dwight and every single other person (my coworkers) for creating MongoDB, cultivating an amazing community and creating a fantastic place to work. • Not in any way to diminish anyone I forgot. Nothing personal, I just have a short attention span and memory!