MongoDB at Gilt (MongoDB Austin 2013)

MongoDB at Gilt
February 15, 2013
Austin Texas

Transcript

1. February 2013 Sean Sullivan @
2. • software engineer • ~2 years at Gilt • work in Gilt’s Portland office • back office applications About me
3. • Gilt Groupe • Gilt’s technology stack • MongoDB at Gilt • Q&A Agenda
4. Gilt Groupe http://www.gilt.com
5. flash sales
6. Everyday at 12 noon
7. what does Gilt sell?
8. Apparel
11. Kids merchandise
13. Home furnishings
15. Food
17. Local deals
19. Travel
22. Gilt technology stack
24. Gilt architecture
25. monolithic application service-oriented architecture 2007 2013
27. 2013 service A service B service C service D service E legacy web app gilt.com
28. Data storage @ Gilt
29. @
30. Why MongoDB? • Ease of use • Horizontal scaling • High availability • Automatic failover
31. Why MongoDB? • Stability • Support • Drivers
32. • MongoDB 2.0 • sharded and non-sharded data • Solid State Drives • MMS for monitoring
33. Use case #1: user profiles
34. • user data in Postgres • legacy Rails app expects to find user data in Postgres • we wanted Gilt’s customer facing applications to retrieve user data from MongoDB Challenges
35. • keep user data in both MongoDB and Postgres • replicate from MongoDB to Postgres Solution
36. Replicating user data user service legacy web app replication service
37. Replication service • listens for RabbitMQ messages ‣ UserCreated message ‣ UserUpdated message • retrieve data using REST API • write data to Postgres using JDBC
38. Use case #2: feature configuration Goal manage the release of new features on gilt.com How feature configuration persisted in MongoDB
39. Rolling out a new feature 1. deploy new application code to production 2. enable feature for Gilt Tech employees 3. ... then enable for all Gilt employees 4. ... then enable for a subset of users 5. gradually ramp up to 100% of users
40. Feature config service Feature configuration gilt.com
41. Use case #3: favorite brands
42. user preference service AJAX Favorite brands
43. Application development
44. mongo-java-driver Morphia Casbah
45. MongoDB Java driver • main class: com.mongodb.Mongo • Mongo object maintains a pool of connections • Latest version: 2.10.1
46. MongoDB Java driver connection pool tuning MongoOptions opts = new MongoOptions(); // example values opts.connectionsPerHost = 50; opts.threadsAllowedToBlockForConnectionMultiplier = 5;
47. com.mongodb.WriteConcern • NORMAL • SAFE • REPLICAS_SAFE • MAJORITY • FSYNC_SAFE • JOURNAL_SAFE
48. Morphia library
49. Morphia • object-document mapper for Java • built on top of mongo-java-driver • map fields using Java annotations
50. // Morphia example import com.google.code.morphia.annotations.*; @Entity(value="features", noClassnameStored = true) public class Feature { @Id ObjectId featureId; @Property("feature_key") @Indexed(unique=true) String featureKey; @Property("release_to_percentage") int releaseToPercentage; }
51. http://code.google.com/p/morphia/
52. Casbah library
53. Casbah • Scala toolkit for MongoDB • built on top of the mongo-java-driver • current version: 2.5.0
54. Casbah • Scala idioms • Scala collections • fluid query syntax
55. https://twitter.com/max4f/status/230503836958199808
56. Gilt Scala code
57. trait MongoWriteConcern { def withSession[T](f: => T)(implicit mongoDb: MongoDB) { mongoDb.requestStart() mongoDb.requestEnsureConnection() try { f } finally { mongoDb.requestDone() } } }
58. Best Practices
59. Best practices Connection tuning explicitly configure Mongo Java Driver connection pool size
60. Best practices Use caution when creating new indexes “creating a new index on a production mongo server can basically cause it to stop working while it builds the index” (source: Gilt production incident report)
61. http://docs.mongodb.org/manual/administration/indexes/#index-building-replica-sets Minimizing impact of building a new index • remove one secondary from replica set • create/rebuild index on this instance • rejoin replica set • repeat on all remaining secondaries • run rs.stepDown() on primary member
62. Best practices use short names for fields to avoid wasted space http://christophermaier.name/blog/2011/05/22/MongoDB-key-names { city: “Portland”, state: “Oregon”, country: “US” } { ci: “Portland”, st: “Oregon”, co: “US” } vs
63. Best practices use explain() during development db.collection.find(query).explain()
64. Best practices use caution when choosing a shard key “It is generally not a good idea to use the default ObjectId as the shard key” source: http://stackoverflow.com/questions/9164356/sharding-by-objectid-is-it-the-right-way
65. Future • shipping addresses • discounts • SKU’s and brands More Gilt data in MongoDB • tag aware sharding Upgrade to MongoDB 2.2
66. Gilt Groupe is hiring! http://techjobs.gilt.com
67. Questions?
68. ssullivan@gilt.com @tinyrobots
69. The end
70. Bonus slides
71. Gilt tech talks • Apache Camel and MongoDB @ Gilt http://bit.ly/OYO37K • Deploying new features @ Gilt http://slidesha.re/OoCYfd • Voldemort @ Gilt http://bit.ly/b9Qhib
72. https://twitter.com/stripe/status/298858032421535744