Dealer.com Versioning System and MongoDB - Kevin Thorley, Dealer.com

MongoBoston 2011

Last August, Dealer.com deployed a new site configuration management system based on MongoDB. We have been running this system in our live production... environment for a little over a year now and have continued to learn and refine our solution. This session will look at our experiences with MongoDB. We will start with the process we used to select MongoDB, the project's lifecycle up until the live deployment, and then what we have learned and changed over the last year. We will also take a look at what our plans are moving forward, and some other places that we have deployed MongoDB within our organization. To give an idea of scale, our current system supports configuration files for over 15,000 customer sites with thousands of revisions per site and an on-disk size of almost 200GB.

Transcript

1. 10/7/11 1 Dealer.com Versioning System Website Configuration Management Built on MongoDB Kevin Thorley Tech Director – Web Solutions Team Dealer.com
2. Who We Are •  Dealer.com •  Founded in 1998 in Burlington, VT •  Leader in Automotive Digital Marketing •  Employs over 500 people in Burlington, VT and Manhattan Beach, CA •  Dealer.com Engineering Team •  Currently hosting close to 15k sites for over 10k clients •  Peak website visitor request load of over 400 req/sec for dynamic content •  Platform is JVM-based •  Java •  Grails •  Spring, Tomcat, Jetty 10/7/11 2
3. 10/7/11 3
4. 10/7/11 4
5. 10/7/11 5
6. 10/7/11 6
7. 10/7/11 7
8. Overall Architecture Akamai Internet Load Balancers CMS Instances GV Services MySQL MongoDB Inv Databases Firewall/Router
9. The Problem Problem Definition Design and implement the next generation of Dealer.com’s website configuration management strategy Requirements •  Support various file formats •  Maintain revision history •  Store the last 6 months of configurations for all sites •  Need to support up to 10x growth after initial deployment •  Provide a client for internal users •  View change logs •  Edit configurations directly Problems with existing solution •  Scalability •  Robustness •  Maintenance 10/7/11 9
10. Proposed Solutions – Java Content Repository Considered three implementations of Java Content Repository (JCR) •  Apache Jackrabbit •  Day CRX •  ModeShape Several issues with these options: •  RDBMS backend solutions were deemed a bad fit •  Non-relational data •  Potentially huge data sets •  SVN backend solution had no support for versioning •  ?!?!? 10/7/11 10
11. Proposed Solutions - MongoDB Our need: A scalable, high performance document store MongoDB provides: “…a scalable, high-performance, open source, document-oriented database.” An obvious choice! Original plan: •  Go live in 3 months with MongoDB 1.4 as the document store •  Keep remainder of application (including cache) in place •  Master/slave deployment with replicated slaves in local environment •  Consider switching to sharded deployment when MongoDB 1.6 available •  Subset of configuration support for first release 10/7/11 11
12. Developing DVS •  Three person development team •  Simplified version control scheme •  Create full copies on each commit •  “MongoDB can handle it” •  MongoDB Java Driver •  Looked at multiple ODM layers •  Decided to build our own ODM •  Uses Java annotations and reflection •  Black box - “Build it, test it, forget it” •  RESTful admin client •  Ruby command line front-end with git-ish commands •  Restlet integration with Mongo backend (DAOs, ODM) 10/7/11 12
13. DVS Architecture 10/7/11 13
14. First Deployment •  Phased deployment over three weeks •  Initial Release (week 1) •  Deploy MongoDB in datacenter •  Initial import of data from previous SVN solution •  Functional Release 1 (week 2) •  Migrate primary site configurations to new DVS system •  Functional Release 2 (week 3) •  Migrate remaining configurations to new DVS system •  Wait and watch •  Decommissioned old svnfs servers •  Rejoice! 10/7/11 14
15. A Year of Updates and Lessons Added additional configuration types over the course of a year •  Added 4 new types between August ‘10 and Jul ’11 •  Most recent addition was site content – a big step! Keeping up with driver versions •  Ran for 9 months with initial driver version •  Appearance of an authorization issue prompted an update to latest Keeping up with MongoDB versions •  Ran for 1 year on version 1.4 •  Upgraded to version 1.8 in August Keeping up with indexes! •  Initial indexes were sufficient for initial data size •  After nine months we realized we needed to re-evaluate our indexes •  An easy fix we should have caught sooner! 10/7/11 15
16. Some Current Stats •  16 Collections (config types) •  Largest collection is site configs •  1847459 documents •  96 GB •  Most sites have << 1000 revisions •  Handful of sites with up to 9000 revisions •  Content collections is small but growing •  691452 •  2.2 GB •  Over 20% growth in 2 months 10/7/11 16
17. Pain Points •  Querying nested maps •  Loading logs for large sites 10/7/11 17
18. The Future Sharded Cluster •  Original plans to migrate to sharded cluster postponed – “It just works” •  Current data size is pointing to the need for a distributed solution Remove the cache? •  Distributed cache is currently a system bottleneck •  Mongo is FAST – do we need the cache? Other Projects •  Small use in another configuration store •  Central part of new ERP system •  In prototyping phase for new inventory metadata project Application Improvements •  Full tagging support •  Draft support One ODM to rule them all! 10/7/11 18
19. Questions? ??? 10/7/11 19
20. 10/7/11 20