Scaling and Flexibility: Two case studies on leveraging the versatility of MongoDB - Erik Kellener, United Future

MongoLA 2012

Two very different MongoDB implementations; both with substantial outcomes. 1) Scaling MongoDB to calculate football team picks, and ranking in realtime, as the results unfold. 2) Leveraging MongoDB's flexible schema to reduce complexity and speed performance for XML data feeds. In both of these examples, we will review the architecture deployed on EC2, and discuss how MongoDB was a crucial component of the optimal solution.

Transcript

1. MongoLA: Scaling & Flexibility January 19th, 2012 Erik Kellener, CTO United Future erik@unitedfuture.com @ekellener Tuesday, January 24, 2012
2. MongoLA: Scaling & Flexibility U N I T E D F U T U R E 2 :: Who are we? Tuesday, January 24, 2012
3. MongoLA: Scaling & Flexibility U N I T E D F U T U R E 2 :: Who are we? UNITED FUTURE CREATES EXPERIENCES THAT COMPEL CONSUMERS TO ACT AND BELIEVE. Tuesday, January 24, 2012
4. MongoLA: Scaling & Flexibility U N I T E D F U T U R E 2 :: Who are we? WEB MOBILE BROADCAST DIGITAL RETAIL SOCIAL MEDIA Tuesday, January 24, 2012
5. MongoLA: Scaling & Flexibility U N I T E D F U T U R E 3 CGD FACEBOOK APP YOKOHAMA TIRE SITE •Football pool - tied to CGD •Enlist & challenge friends •Rank with celebrities :: Case Studies Tuesday, January 24, 2012
6. MongoSV: Scaling & Flexibility U N I T E D F U T U R E 4 :: CGD Facebook App:: How it works Each week, users enter their team picks for Saturday games. Tuesday, January 24, 2012
7. MongoSV: Scaling & Flexibility U N I T E D F U T U R E 5 :: CGD Facebook App: How it works. Stats. reported by week and by season Tuesday, January 24, 2012
8. MongoSV: Scaling & Flexibility U N I T E D F U T U R E 6 :: CGD Facebook App:: How it works Ranking compared and reported by : •FB friends •Celebrities •Custom Groups •Overall participants. Tuesday, January 24, 2012
9. MongoLA: Scaling & Flexibility U N I T E D F U T U R E 7 ::CGD :: Distribution Stack EC2 (LG) ELB PHP CodeIgniter Doctrine APC (PECL) MongoDB 1 - Mongod AWS Architecture Cherokee Linux EBS EC2 (LG) Cherokee Linux EBS EC2 (LG) Auto Scale (max 9) MongoDB Linux EBS EC2 (LG) MongoDB Linux EBS EC2 (LG) MongoDB Linux EBS EC2 (LG) Application Systems Replica Set Arbiter 2 - Mongod Arbiter 3 - Mongod 1 2 3 Tuesday, January 24, 2012
10. MongoLA: Scaling & Flexibility U N I T E D F U T U R E 7 ::CGD :: Distribution Stack EC2 (LG) ELB PHP CodeIgniter Doctrine APC (PECL) MongoDB 1 - Mongod AWS Architecture Cherokee Linux EBS EC2 (LG) Cherokee Linux EBS EC2 (LG) Auto Scale (max 9) MongoDB Linux EBS EC2 (LG) MongoDB Linux EBS EC2 (LG) MongoDB Linux EBS EC2 (LG) Application Systems Replica Set Arbiter 2 - Mongod Arbiter 3 - Mongod 1 2 3 Tuesday, January 24, 2012
11. MongoLA: Scaling & Flexibility U N I T E D F U T U R E 7 ::CGD :: Distribution Stack EC2 (LG) ELB PHP CodeIgniter Doctrine APC (PECL) MongoDB 1 - Mongod AWS Architecture Cherokee Linux EBS EC2 (LG) Cherokee Linux EBS EC2 (LG) Auto Scale (max 9) MongoDB Linux EBS EC2 (LG) MongoDB Linux EBS EC2 (LG) MongoDB Linux EBS EC2 (LG) Application Systems Replica Set Arbiter 2 - Mongod Arbiter 3 - Mongod 1 2 3 Tuesday, January 24, 2012
12. MongoLA: Scaling & Flexibility U N I T E D F U T U R E 8 :: CGD :: ODM :: Doctrine Tuesday, January 24, 2012
13. MongoLA: Scaling & Flexibility U N I T E D F U T U R E 8 :: CGD :: ODM :: Doctrine •Doctrine : Object Data Mapper (ODM) •Great abstraction layer. •Specification doesn’t require separate file (e.g. yaml,xml) •Transactional write-behind •Powerful persistence behaviors. (prepersist, postpersist) Tuesday, January 24, 2012
14. MongoLA: Scaling & Flexibility U N I T E D F U T U R E 8 :: CGD :: ODM :: Doctrine •Doctrine : Object Data Mapper (ODM) •Great abstraction layer. •Specification doesn’t require separate file (e.g. yaml,xml) •Transactional write-behind •Powerful persistence behaviors. (prepersist, postpersist) Example: Document Specification <?php namespace Documents; use Doctrine\Common\Collections\ArrayCollection; /** @Document(db="saturdayselections", collection="users") */ class User { /** @Id */ private $id; /** @Date */ private $created; /** @String @Index(order="asc") */ private $name; /** @EmbedOne(targetDocument="FacebookUser") */ private $facebook; /** @EmbedMany(targetDocument="UserPick") */ private $picks; .... Tuesday, January 24, 2012
15. MongoSV: Scaling & Flexibility U N I T E D F U T U R E 9 :: CGD :: ODM :: Doctrine •Doctrine : Object Data Mapper (ODM) •Great abstraction layer. •Specification doesn’t require separate file (e.g. yaml,xml) •Transactional write-behind •Powerful persistence behaviors. (prepersist, postpersist) Tuesday, January 24, 2012
16. MongoSV: Scaling & Flexibility U N I T E D F U T U R E 9 :: CGD :: ODM :: Doctrine •Doctrine : Object Data Mapper (ODM) •Great abstraction layer. •Specification doesn’t require separate file (e.g. yaml,xml) •Transactional write-behind •Powerful persistence behaviors. (prepersist, postpersist) Example: Corresponding MongoDB shell query. ... db.user.find({“facebook.id” : {“$in” : [“102929”,”20293903”]}}).limit(5); ... Tuesday, January 24, 2012
17. MongoSV: Scaling & Flexibility U N I T E D F U T U R E 9 :: CGD :: ODM :: Doctrine •Doctrine : Object Data Mapper (ODM) •Great abstraction layer. •Specification doesn’t require separate file (e.g. yaml,xml) •Transactional write-behind •Powerful persistence behaviors. (prepersist, postpersist) Example: Doctrine Query ... $ids = array(); $result = $this->doctrine->createQuery('Documents\User') ->select('id') ->field('facebook.id')->in((array) $facebook_ids) ->limit($limit) ->hydrate(FALSE) ->getQuery() ->execute() ->toArray(); ... Example: Corresponding MongoDB shell query. ... db.user.find({“facebook.id” : {“$in” : [“102929”,”20293903”]}}).limit(5); ... Tuesday, January 24, 2012
18. MongoLA: Scaling & Flexibility U N I T E D F U T U R E 10 •Design considerations: •User centered model •“Picks” embedded array. •“weekly” embedded subdoc. (initialize first) ::CGD :: Data Schema :: Users Collection OverallResult facebook weekly UserPick overall WeeklyResult FacebookUser picks name id Model: users Users name id Model: FacebookUser facebook points outcome week game_id team Model: UserPick has_many_and_belongs_to_m any :teams picks percentage total incorrect correct Model: OverallResult has_many_and_belongs_to_m any :teams overall week total incorrect correct Model: WeeklyResult weekly Tuesday, January 24, 2012
19. MongoLA: Scaling & Flexibility U N I T E D F U T U R E 11 •Design considerations: •“Groups” embeds members by status (active, banned, pending) •“Groups” contains an array of tags for keyword searching. • “Game” embeds Home & Away teams along with stats & ranks. ::CGD :: Data Schema :: Groups & Game Collection away id winner week Team home Team Model: Game Game record rank name Model: Team Team status GroupMembers id tags members array[] Model: Groups Groups array[] pending banned array[] active name array[] Model: groupmembers GroupMembers Tuesday, January 24, 2012
20. MongoLA: Scaling & Flexibility U N I T E D F U T U R E 12 :: College GameDay Facebook App:: Key learnings Tuesday, January 24, 2012
21. MongoLA: Scaling & Flexibility U N I T E D F U T U R E 12 :: College GameDay Facebook App:: Key learnings •Key learnings •If using EC2, ensure replica sets are running in multiple zones. •Don’t worry about data normalization, embed subdocs when possible •Use array subdocs when: •Highest flexibility in queries. •Filtering on the client doesn’t pose a problem. •Updates are not limited by positional operator •Improve query performance by initializing data (size/type) upon initial insert. •ODM will make your life much easier. Tuesday, January 24, 2012
22. MongoSV: Scaling & Flexibility U N I T E D F U T U R E 13 CGD FACEBOOK APP •Product research •Consumer focused •Mix of CMS content, and feeds YOKOHAMA TIRE SITE :: Case Studies Tuesday, January 24, 2012
23. MongoSV: Scaling & Flexibility U N I T E D F U T U R E 13 •Product research •Consumer focused •Mix of CMS content, and feeds YOKOHAMA TIRE SITE :: Case Studies Tuesday, January 24, 2012
24. MongoLA: Scaling & Flexibility U N I T E D F U T U R E 14 :: Yokohama Tire Site: Case Study #2: Yokohama Tire Content Mgmt System Car Catalog Product/Tire Catalog Dealer Locator Tuesday, January 24, 2012
25. MongoLA: Scaling & Flexibility U N I T E D F U T U R E 15 :: Yokohama Site::Distribution and Data •Running in EC2 (similar to CGD setup) •Application Stack PHP CodeIgniter Doctrine MySQL (PECL) MongoDB Expression Engine TIRE FEED (NIGHTLY) <?xml version="1.0"?> <!--Created by PRO_PTS_PRT_SELECTOR--> <!--Create date: 2011/01/18--> <PART_LIST> <PART> <PART_NO>10191</PART_NO> <PROD_LINE>HPT</PROD_LINE> <TRD_DES>SPORT</TRD_DES> <SIZE>275/45R19</SIZE> <UTQG>280/AA/A</UTQG> <COMPOUND/> <PLY_RATING/> <PLY/> <LOAD_INDEX>108</LOAD_INDEX> <SPEED_RATING>Y</SPEED_RATING> <SPEED_RT_NUMERIC>12</SPEED_RT_NUMERIC> <TREAD_DEPTH>10</TREAD_DEPTH> <SDWL_TRA>BW</SDWL_TRA> <SHIP_WT>35.05</SHIP_WT> <OEM_PART>1</OEM_PART> <RIM_SIZE>19</RIM_SIZE> <ASPECT_RATIO>45</ASPECT_RATIO> <SECT_WDTH>275</SECT_WDTH> <STS>A</STS> <BRAND>Y</BRAND> <LOAD_RANGE>XL</LOAD_RANGE> <TREAD_TYPE/> <PUBLISHED_PART_DESC>275/45R19 108Y XL</PUBLISHED_PART_DESC> <MARKETING_TREAD_NAME>ADVAN SPORT</MARKETING_TREAD_NAME> <LAST_OE_YEAR>2011</LAST_OE_YEAR> <OE_APPLICATIONS> <OE_APP> <OE_VEH_MAKE>PORSCHE</OE_VEH_MAKE> <OE_VEH_MODEL>CAYENNE</OE_VEH_MODEL> <OE_YEAR>2011</OE_YEAR> </OE_APP> <OE_APP> <OE_VEH_MAKE>VOLKSWAGEN</OE_VEH_MAKE> <OE_VEH_MODEL>TOUAREG</OE_VEH_MODEL> <OE_YEAR>2010</OE_YEAR> </OE_APP> </OE_APPLICATIONS> </PART> </PART_LIST> DISTRIBUTION Tuesday, January 24, 2012
26. MongoLA: Scaling & Flexibility U N I T E D F U T U R E 16 Design considerations: •“tires” is a nightly XML feed of products. •Feed structure may change without notice. •“tireclass” separately maintained CSV for classifying “tires” • :: Yokohama Site::Data :: Tires Collection name style array array application size array category vehicle_type id array Model: tires tires array vehicle_type array application array style name Model: tire_classification tireclass Product/Tire Catalog Example query tire after tireclass has been merged. ... $query = $this->doctrine->createQuery('Documents\Tire') ->field('category.vehicle_type')->in((array) $vehicle_type) ->field('category.business_type')->equals($type); $this->_exclude_odd_tires($query); $result = $query->getQuery()->execute(); ... Tuesday, January 24, 2012
27. MongoLA: Scaling & Flexibility U N I T E D F U T U R E 17 Design considerations •Nightly XML feed of dealers. •Structure may change without notice. •New dealers are geocoded on import. •Leverages MongoDB spatial queries. :: Yokohama Site::Data :: Dealers Collection coordinates location id Model: dealer_locations Dealer_locations lon lat Model: coordinates coordinates Dealer Locator Example : Geospatial query for Lat/lon on Dealers. <?php namespace Documents; /** * @Document(db="yokohama", collection="dealer_locations") */ class DealerLocation { $bus_unit = $search_commercial ? self::COMMERCIAL_CODE : self::CONSUMER_CODE; $query = $this->doctrine->createQuery('Documents\Dealer') ->field('coordinates')->withinCenter($center_lat, $center_lon, $radius)->field('bus_unit')->equals($bus_unit); ... return $query->sort('distance', 'asc') ->limit(26) ->getQuery() ->execute(); } Tuesday, January 24, 2012
28. MongoLA: Scaling & Flexibility U N I T E D F U T U R E 18 :: Yokohama Site::Data :: Cars Collection Design considerations: •Separately maintained CSV. •Heavy remapping of fields. •Imported into MongoDB size_desc id make model year Model: cars cars Car Catalog <?php namespace Documents; /** * @Document(db="yokohama", collection="cars") * @HasLifecycleCallbacks */ class Car { /** @Id */ private $id; /** @String */ private $size_desc; /** @String @Index(order="asc") */ private $speed_rating; /** @PrePersist */ public function parse_speed_rating() { // @TODO handle this style -> LT235/85R16/E if (empty($this->speed_rating)) { $slash_pos = strpos($this->size_desc, '/'); $speed_rating = substr($this->size_desc, $slash_pos + 3, 1); // alpha letter but not R if ((preg_match( '/[A-QS-Z]/i', $speed_rating))) {$this->set_speed_rating($speed_rating);} } $this->set_speed_rating_numeric(); return; } Tuesday, January 24, 2012
29. MongoLA: Scaling & Flexibility U N I T E D F U T U R E 19 :: Yokohama Site::Data :: Key learnings •Key learnings •MongoDB’s flexible schema is a perfect fit for XML data imports. Using RDBMS would be far more complex. •When dealing with “unscrubbed” data, Doctrine comes to the rescue with “PrePersist” behaviors. •MongoDB’s geospatial support can be handy for location based data. •And yes, MongoDB rocks!! Tuesday, January 24, 2012
30. Tuesday, January 24, 2012
31. Questions? Tuesday, January 24, 2012
32. erik@unitedfuture.com / @ekellener Tuesday, January 24, 2012