Upgrade to Pro — share decks privately, control downloads, hide ads and more …

MongoDB DC 2012: "Operationalizing" MongoDB At AOL

D8fc2580cfaca035f666d9e4ee79a7f7?s=47 mongodb
June 26, 2012
270

MongoDB DC 2012: "Operationalizing" MongoDB At AOL

Michael DelNegro, AOL
Learn about the challenges of bringing MongoDB into a large organization with a centralized operations group. "Operationalizing" consists of establishing and communicating installation, setup, monitoring, alarming standards and overall best practices.

D8fc2580cfaca035f666d9e4ee79a7f7?s=128

mongodb

June 26, 2012
Tweet

Transcript

  1. Michael DelNegro! Principal Database Administrator! AOL" “Operationalizing” MongoDB@AOL"

  2. About Me" •  DBA at AOL (Dulles) for 6+ years"

    •  Original DBMS Background is in Sybase" •  Now MySQL, MongoDB, PostgreSQL and NoSQL (Neo4J, Redis, etc)" •  Check out www.meetup.com/NOVA-MySQL" "
  3. The First Mongo at AOL" •  A MapQuest Developer in

    the Summer of 2010 Replaced a MySQL Datastore with Mongo" •  That Developer left in early 2011 :-/" •  Was Handed Over to Operations and Yours Truly "
  4. Never Forget Your First" •  Not Small: 40+ Hosts" • 

    Upgrade Challenges" •  Connection Floods" •  Support to the Rescue" •  Magnified the Reasons to Operationalize"
  5. Operation “Operationalize”"

  6. Operation “Operationalize”" •  Take Advantage of 10Gen Support" •  Establish

    Standards" •  Monitoring/Alerting" •  Backups" •  Information Sharing" •  Challenges"
  7. 10Gen Support" •  JIRA Documents Cases Enabling Information Sharing and

    Referencing" •  Serves Us Well in Both “Emergencies” and “Hey Quick Question”" •  Training Classes: Administration & Development"
  8. Standard Host Setup" •  CentOS 6" •  XFS/EXT4" •  Deadline

    I/O Scheduler" •  dirty_ratio = 10 (down from 40)" •  dirty_background_ratio = 5 (down from 10)" •  noatime" "
  9. Standard Build Scripts" •  Use 64bit Linux Binaries Stored in

    AOL Repository" •  mongodb_install.sh" – Parameters Include:" •  MongoDB Version" •  Replica Set Name" •  Sharded? (Y/N)" •  Port (27017 is standard)" •  Type (DB, Arbiter, Config, Mongos)"
  10. Standard'Directory/Layouts' •  Based'on'our'standard'MySQL'layout' •  Binaries'in'/opt/local/pkg' •  /db/mongodb/<port>'base'directory' –  Ex.'/db/mongodb/27017' – 

    Can'Override' •  /db/mongodb/27017Grepset1' •  Under'base'directory' –  data' –  etc'(config'files)' –  logs' –  scripts'(ex'stop/start)'
  11. ORB" •  AOL Technologies’ Configuration Management Database (CMDB)" •  Integrated

    with many authoritative data repositories" •  Unique namespace for many operations data points" •  Data model for operations management" •  Projects, Assets, People, Applications, Network Data, HCM" •  SQL Interface"
  12. Monitoring/Alerting" " •  Argus" •  Nagios" •  Component Test Tool"

    •  MMS (MongoDB Monitoring Service)" "
  13. Argus" •  Metric and Event" – Collector" – Thresholds " – Management" – Data

    Viewing" '
  14. Argus" •  Capture serverStatus Metrics (once a minute)" •  Capture

    Replication Lag" •  Great for Trending" •  Great for Capacity Planning" •  Great for Troubleshooting" •  We Also Use for Host Metrics (CPU, I/O, etc)" •  Administrated By a Small and very Busy Group" –  Requested Additions/Changes Can Be Slow"
  15. Nagios" •  Great For Fault-detection/Alerting" •  Great For Show Me

    What Is Currently Broken" •  Great For Service Availability Metrics" •  Flexible" •  Reduces Pressure On NOC" •  Integrates With Netcool, Ignore Tool" •  We Write Our Own Plug-Ins"
  16. Component Test Tool" •  Runs Parallel HTTP GETs Across a

    Group of Hosts" •  Matches Regular Expressions On the Page Content (or HTTP Headers)" •  Displays Number (or Content) of Matches in a HTML Table"
  17. MMS"

  18. Backups" •  Scripts Delivered With Install" •  Choice Between LVM

    (default) and Standard (mongodump)" •  Ability to Backup Sharded Setup" '
  19. Mongo @ AOL Currently" ' '

  20. Mongo @ AOL Currently" •  30+ Active MongoDB Projects" • 

    500+ Servers in Production" •  Developed in:" –  Java" –  PHP" –  Ruby" –  C#" –  Erlang" –  Scala" –  Perl" –  C++"
  21. Information Sharing" •  Internal Wikis/Websites" •  UnUniversity (Organized Brown Bags)"

    •  Architecture Reviews" •  10Gen JIRA Access'
  22. Challenges" •  Developers Love It " •  BUT Requires More

    Upfront Planning Than They May Realize" •  Should NOT Be Our Default Datastore" •  Better Developer OnBoarding" •  Aggressive Upgrading Is Required" •  Coordination of Driver Upgrades"
  23. The Future" •  “MongoDB As a Service”" •  Puppet or

    Chef" •  Internal Administration Console" – Already has Visual Topologies" •  Internal MongoDB Operations Class" – First One is in July"
  24. Resources" •  Recommended Presentation: Todd Dampier’s Rock-Solid Mongo Ops (mongolab)"

    •  Check out Jon Reed’s MongoDB for Online Advertising at AOL (MongoDB UK 2012)" •  www.meetup.com/Washington-DC- MongoDB-Users-Group/"
  25. Thank You!" •  www.slideshare.net/radiocats" •  @radiocats" •  www.linkedin.com/in/mdelnegro" •  Thank

    you, SeniorGif"