MongoDB DC 2012: "Operationalizing" MongoDB At AOL

Michael DelNegro, AOL
Learn about the challenges of bringing MongoDB into a large organization with a centralized operations group. "Operationalizing" consists of establishing and communicating installation, setup, monitoring, alarming standards and overall best practices.

Transcript

1. Michael DelNegro! Principal Database Administrator! AOL" “Operationalizing” MongoDB@AOL"
2. About Me" •  DBA at AOL (Dulles) for 6+ years" •  Original DBMS Background is in Sybase" •  Now MySQL, MongoDB, PostgreSQL and NoSQL (Neo4J, Redis, etc)" •  Check out www.meetup.com/NOVA-MySQL" "
3. The First Mongo at AOL" •  A MapQuest Developer in the Summer of 2010 Replaced a MySQL Datastore with Mongo" •  That Developer left in early 2011 :-/" •  Was Handed Over to Operations and Yours Truly "
4. Never Forget Your First" •  Not Small: 40+ Hosts" •  Upgrade Challenges" •  Connection Floods" •  Support to the Rescue" •  Magnified the Reasons to Operationalize"
5. Operation “Operationalize”"
6. Operation “Operationalize”" •  Take Advantage of 10Gen Support" •  Establish Standards" •  Monitoring/Alerting" •  Backups" •  Information Sharing" •  Challenges"
7. 10Gen Support" •  JIRA Documents Cases Enabling Information Sharing and Referencing" •  Serves Us Well in Both “Emergencies” and “Hey Quick Question”" •  Training Classes: Administration & Development"
8. Standard Host Setup" •  CentOS 6" •  XFS/EXT4" •  Deadline I/O Scheduler" •  dirty_ratio = 10 (down from 40)" •  dirty_background_ratio = 5 (down from 10)" •  noatime" "
9. Standard Build Scripts" •  Use 64bit Linux Binaries Stored in AOL Repository" •  mongodb_install.sh" – Parameters Include:" •  MongoDB Version" •  Replica Set Name" •  Sharded? (Y/N)" •  Port (27017 is standard)" •  Type (DB, Arbiter, Config, Mongos)"
10. Standard'Directory/Layouts' •  Based'on'our'standard'MySQL'layout' •  Binaries'in'/opt/local/pkg' •  /db/mongodb/<port>'base'directory' –  Ex.'/db/mongodb/27017' –  Can'Override' •  /db/mongodb/27017Grepset1' •  Under'base'directory' –  data' –  etc'(config'files)' –  logs' –  scripts'(ex'stop/start)'
11. ORB" •  AOL Technologies’ Configuration Management Database (CMDB)" •  Integrated with many authoritative data repositories" •  Unique namespace for many operations data points" •  Data model for operations management" •  Projects, Assets, People, Applications, Network Data, HCM" •  SQL Interface"
12. Monitoring/Alerting" " •  Argus" •  Nagios" •  Component Test Tool" •  MMS (MongoDB Monitoring Service)" "
13. Argus" •  Metric and Event" – Collector" – Thresholds " – Management" – Data Viewing" '
14. Argus" •  Capture serverStatus Metrics (once a minute)" •  Capture Replication Lag" •  Great for Trending" •  Great for Capacity Planning" •  Great for Troubleshooting" •  We Also Use for Host Metrics (CPU, I/O, etc)" •  Administrated By a Small and very Busy Group" –  Requested Additions/Changes Can Be Slow"
15. Nagios" •  Great For Fault-detection/Alerting" •  Great For Show Me What Is Currently Broken" •  Great For Service Availability Metrics" •  Flexible" •  Reduces Pressure On NOC" •  Integrates With Netcool, Ignore Tool" •  We Write Our Own Plug-Ins"
16. Component Test Tool" •  Runs Parallel HTTP GETs Across a Group of Hosts" •  Matches Regular Expressions On the Page Content (or HTTP Headers)" •  Displays Number (or Content) of Matches in a HTML Table"
17. MMS"
18. Backups" •  Scripts Delivered With Install" •  Choice Between LVM (default) and Standard (mongodump)" •  Ability to Backup Sharded Setup" '
19. Mongo @ AOL Currently" ' '
20. Mongo @ AOL Currently" •  30+ Active MongoDB Projects" •  500+ Servers in Production" •  Developed in:" –  Java" –  PHP" –  Ruby" –  C#" –  Erlang" –  Scala" –  Perl" –  C++"
21. Information Sharing" •  Internal Wikis/Websites" •  UnUniversity (Organized Brown Bags)" •  Architecture Reviews" •  10Gen JIRA Access'
22. Challenges" •  Developers Love It " •  BUT Requires More Upfront Planning Than They May Realize" •  Should NOT Be Our Default Datastore" •  Better Developer OnBoarding" •  Aggressive Upgrading Is Required" •  Coordination of Driver Upgrades"
23. The Future" •  “MongoDB As a Service”" •  Puppet or Chef" •  Internal Administration Console" – Already has Visual Topologies" •  Internal MongoDB Operations Class" – First One is in July"
24. Resources" •  Recommended Presentation: Todd Dampier’s Rock-Solid Mongo Ops (mongolab)" •  Check out Jon Reed’s MongoDB for Online Advertising at AOL (MongoDB UK 2012)" •  www.meetup.com/Washington-DC- MongoDB-Users-Group/"
25. Thank You!" •  www.slideshare.net/radiocats" •  @radiocats" •  www.linkedin.com/in/mdelnegro" •  Thank you, SeniorGif"