MongoDB at Gilt (MongoDB Portland 2013)

MongoDB at Gilt
MongoDB Portland
April 30, 2013

Transcript

1. April 2013 Sean Sullivan @
2. • software engineer • ~2 years at Gilt • work in Gilt’s Portland office • back office applications About me
3. • Gilt Groupe • Gilt’s technology stack • MongoDB at Gilt • Q&A Agenda
4. Gilt Groupe http://www.gilt.com
5. flash sales
6. Everyday at 12 noon
7. what does Gilt sell?
13. Gilt technology stack
15. Gilt architecture
16. monolithic application service-oriented architecture 2007 2013
18. 2013 service A service B service C service D service E legacy web app gilt.com
19. Data storage @ Gilt
20. @
21. Why MongoDB? • Ease of use • Horizontal scaling • High availability • Automatic failover
22. Why MongoDB? • Stability • Support • Drivers
23. • MongoDB 2.0 • sharded and non-sharded data • Solid State Drives • MMS for monitoring
24. Use case #1: user profiles
25. • user data in Postgres • legacy Rails app expects to find user data in Postgres • we wanted Gilt’s customer facing applications to retrieve user data from MongoDB Challenges
26. • keep user data in both MongoDB and Postgres • replicate from MongoDB to Postgres Solution
27. Replicating user data user service legacy web app replication service
28. Replication service • listens for RabbitMQ messages ‣ UserCreated message ‣ UserUpdated message • retrieve data using REST API • write data to Postgres using JDBC
29. Use case #2: feature configuration Goal manage the release of new features on gilt.com How feature configuration persisted in MongoDB
30. Rolling out a new feature 1. deploy new application code to production 2. enable feature for Gilt Tech employees 3. ... then enable for all Gilt employees 4. ... then enable for a subset of users 5. gradually ramp up to 100% of users
31. Feature config service Feature configuration gilt.com
32. Use case #3: favorite brands
33. user preference service AJAX Favorite brands
34. Application development
35. mongo-java-driver Morphia Casbah
36. https://twitter.com/max4f/status/230503836958199808
37. Gilt Scala code
38. trait MongoWriteConcern { def withSession[T](f: => T)(implicit mongoDb: MongoDB) { mongoDb.requestStart() mongoDb.requestEnsureConnection() try { f } finally { mongoDb.requestDone() } } }
39. Best Practices
40. Best practices Connection tuning explicitly configure Mongo Java Driver connection pool size
41. Best practices Use caution when creating new indexes “creating a new index on a production mongo server can basically cause it to stop working while it builds the index” (source: Gilt production incident report)
42. http://docs.mongodb.org/manual/administration/indexes/#index-building-replica-sets Minimizing impact of building a new index • remove one secondary from replica set • create/rebuild index on this instance • rejoin replica set • repeat on all remaining secondaries • run rs.stepDown() on primary member
43. Best practices use short names for fields to avoid wasted space http://christophermaier.name/blog/2011/05/22/MongoDB-key-names { city: “Portland”, state: “Oregon”, country: “US” } { ci: “Portland”, st: “Oregon”, co: “US” } vs
44. Best practices use explain() during development db.collection.find(query).explain()
45. Best practices use caution when choosing a shard key “It is generally not a good idea to use the default ObjectId as the shard key” source: http://stackoverflow.com/questions/9164356/sharding-by-objectid-is-it-the-right-way
46. Future • shipping addresses • SKU’s • brands More Gilt data in MongoDB • text search • hashed shard keys Upgrade to MongoDB 2.4
47. Gilt Groupe is hiring! http://techjobs.gilt.com
48. Questions?
49. ssullivan@gilt.com @tinyrobots
50. The end
51. Bonus slides
52. Gilt tech talks • Apache Camel and MongoDB @ Gilt http://bit.ly/OYO37K • Deploying new features @ Gilt http://slidesha.re/OoCYfd • Voldemort @ Gilt http://bit.ly/b9Qhib
53. https://twitter.com/stripe/status/298858032421535744