Experiences using ReactiveMongo to build a JSON API with Play

After ReactiveMongo was released late last year, I spent 2 months rewriting an application in Scala to use it. It is as close to the MongoDB wire-protocol as possible but unlike the official drivers, it is fully asynchronous & non-blocking, using the powerful Akka actors underneath the hood.

This talk will compare some of the idiomatic differences between programming with different MongoDB drivers, architectural patterns and also gives examples of the performance implications of these decisions.

After the talk, the attendees should have a clear understanding of why the concepts behind ReactiveMongo are important and also be able to reason about how they structure their application.

Transcript

1. ReactiveMongo “Experiences using ReactiveMongo to build a JSON API with Play” - Alexander Jarvis London, 9th April 2013 PANAXIOM
2. Overview • A brief history & context • What is ReactiveMongo? • Asynchronous & Non-blocking I/O • Rewrite, reduce. • Vanity benchmarks • Typical usage & patterns
3. ๏ Real-time location sharing ๏ Currently in alpha ๏ API available http://meep.io ๏ Uses ReactiveMongo
4. Common project goals • Provide a fast JSON API over HTTP • High throughput, low latency • Scalability curl https://meep.io/users/UR6pdkqrFEgAG_H2 -H 'Authorization: Bearer xd97JMba+a*yM51ZsBEUUT)_fgYt10BO' -H 'Accept: application/json' { "id":"UR6pdkqrFEgAG_H2", "email":"charles@babbage.com", "firstName":"Charles", "lastName":"Babbage", "image":"UR6qFEqrFBsA3OL9", "updated":"2013-02-15T22:32:38.410+01:00" }
5. • Web framework for Java and Scala • 1.0 public 2009 (mostly Java, Scala module) • 2.0 rewritten in Scala at end of 2011 • Gaining momentum, support from Typesafe • Open Source (Apache 2) playframework.com
6. • Stateless (low memory, horizontal scaling) • Non-blocking I/O • Built using the Akka Actor Model & Netty
7. Enough Playing around, more mongo • Transition from relational databases and ORMs to MongoDB and documents makes sense for a JSON API • With de-normalised data a JSON API is even closer to representations in the database Web Client Web Service MongoDB BSON JSON
8. Finding your MongoDB Nirvana Official Java driver (with mongo-jackson-mapper) public boolean updateAccessToken() { ! return db().update(DBQuery.is("_id", new org.bson.types.ObjectId(id)), ! ! ! new BasicDBObject("$set", new BasicDBObject("accessToken", accessToken)), ! ! ! false, ! ! ! false, ! ! ! WriteConcern.SAFE).getLastError().ok(); }
9. Finding your MongoDB Nirvana Official Java driver (with mongo-jackson-mapper) public boolean updateAccessToken() { ! return db().update(DBQuery.is("_id", new org.bson.types.ObjectId(id)), ! ! ! new BasicDBObject("$set", new BasicDBObject("accessToken", accessToken)), ! ! ! false, ! ! ! false, ! ! ! WriteConcern.SAFE).getLastError().ok(); } public boolean updateAccessToken() { ! return users().update("{_id: #}", this._id).concern(WriteConcern.SAFE) ! ! ! .with("{$set: {accessToken: #}}", this.accessToken).getLastError().ok(); } Jongo after 1 month
10. ReactiveMongo • Asynchronous & Non-Blocking Scala Driver for MongoDB • Written by Stephane Godbillion at Zenexity (original company behind Play) • Released 30th August 2012 • Open Source (Apache 2) reactivemongo.org
11. Motivation • Classic synchronous DB drivers block threads until response is received • Threads can only scale up to a limit • Context switches are expensive • Made specifically for Play • The future is streaming data • Related to the “The C10K problem”
12. Non-blocking I/O The fast food restaurant analogy: non-blocking: “When you place an order at the cashier, it’s sent off for someone else to fulfil and you step aside for the next customer to be served.” blocking: “When you place an order at the cashier, you wait for your order and no other customers can be served...”
13. Scala Futures • A composable container for the result of a computation that may not exist yet. • ReactiveMongo’s basic Cursor functions return Futures: def headOption(): Future[Option[T]] def toList(upTo: Int): Future[List[T]] def toList(): Future[List[T]] (simplified)
14. Rewrite, reduce • Strategic, long-term decision • Play’s Java API didn’t feel as fluid as 1.0
15. Rewrite, reduce • Strategic, long-term decision • Play’s Java API didn’t feel as fluid as 1.0 24th Sep -> 7th Nov 2012 (1 month and 14 days) 0 525 1,050 1,575 2,100 Lines of code 1,199 2,028 Java Scala
16. Vanity benchmarks Downloading a GridFS file 0 375 750 1125 1500 244 Kb file 1464.11 966.68 Requests / sec (mean) Official Reactive Apache Bench ab -n 5000 -c 10
17. Vanity benchmarks Downloading a GridFS file 0 375 750 1125 1500 244 Kb file 1464.11 966.68 Requests / sec (mean) Official Reactive 0 23 45 68 90 6.2 Mb file 86.12 57.12 Requests / sec (mean) Official Reactive Apache Bench ab -n 5000 -c 10
18. Vanity benchmarks 33% increase in requests / second Official Java Driver ReactiveMongo • Noticed fans spinning when using the Java Driver • ReactiveMongo handles resources better
19. Vanity benchmarks 33% increase in requests / second Official Java Driver ReactiveMongo • Noticed fans spinning when using the Java Driver • ReactiveMongo handles resources better
20. A Simple Example Play Routes File https://github.com/alexanderjarvis/reactivemongo-json-demo
21. Mapping the Future Inside a Play Controller
22. Mapping the Future Inside a Play Controller Future[List[Thing]]
23. Mapping the Future Inside a Play Controller { async Future[List[Thing]]
24. GET a /thing curl localhost:9000/things/5143294a8a147dbf12ad65bb { "id":"5143294a8a147dbf12ad65bb", "title":"the same thing", "description":"updated thing", "created":1363355978000 }
25. GET a /thing curl localhost:9000/things/5143294a8a147dbf12ad65bb { "id":"5143294a8a147dbf12ad65bb", "title":"the same thing", "description":"updated thing", "created":1363355978000 } Future[Option[Thing]]
26. Future is a Monad • Allows you to chain results together • map, flatMap, filter • flatMap will flatten the result e.g. two nested maps create a Future of a Future. Future[Future[Result]] Future[Result] flatMap
27. Model • Manually define the reader/writer • Scala macros will reduce this to one line • Similar to Json reads writes combinators
28. Json implicit conversions (from a url shortener)
29. Json implicit conversions (from a url shortener) Json path
30. 3 Ways to structure your web app
31. 1. Monolithic Controller • What we have seen thus far • Mongo collection as val in Controller • Choice of JSON to BSON implicit conversion • Will get messy when mixing more models and controllers
32. DAO Model Controller Data Source 2. Data Access Objects Abstraction of common operations, separate from model
33. DAO Model Controller Data Source 2. Data Access Objects Abstraction of common operations, separate from model
34. DAO and flatMap POST /meetings/:id/accept
35. DAO and flatMap 1 POST /meetings/:id/accept
36. DAO and flatMap 1 2 POST /meetings/:id/accept
37. 3. JSON Coast-to-Coast • Validate and transform JSON without static case class models • Data flow • JSON to BSON implicit conversions • Business logic needs to speak JSON github.com/mandubian/play2-json-demo example at
38. Enumerators / Iteratees • Streaming data • Immutable Producer / Consumer Potentially huge list in memory Streamed from Mongo
39. Try it out!
40. References 1. Play framework http://playframework.com/ 2. Mongo Jackson Mapper http://mongojack.org/ 3. Jongo http://jongo.org/ 4. ReactiveMongo http://reactivemongo.org/ 5. The C10K problem http://www.kegel.com/c10k.html 6. Scala Futures http://docs.scala-lang.org/overviews/core/futures.html 7. Scala Futures (akka) http://doc.akka.io/docs/akka/2.1.1/scala/futures.html 8. Reactive-json-demo https://github.com/alexanderjarvis/reactivemongo-json-demo 9. Linky https://github.com/alexanderjarvis/linky 10. JSON Coast-to-Coast http://mandubian.com/2013/01/13/JSON-Coast-to-Coast/