MongoDB DC 2012: "Operationalizing" MongoDB At AOL

Slide 1

Slide 1 text

Michael DelNegro! Principal Database Administrator! AOL" “Operationalizing” MongoDB@AOL"

Slide 2

Slide 2 text

About Me" •  DBA at AOL (Dulles) for 6+ years" •  Original DBMS Background is in Sybase" •  Now MySQL, MongoDB, PostgreSQL and NoSQL (Neo4J, Redis, etc)" •  Check out www.meetup.com/NOVA-MySQL" "

Slide 3

Slide 3 text

The First Mongo at AOL" •  A MapQuest Developer in the Summer of 2010 Replaced a MySQL Datastore with Mongo" •  That Developer left in early 2011 :-/" •  Was Handed Over to Operations and Yours Truly "

Slide 4

Slide 4 text

Never Forget Your First" •  Not Small: 40+ Hosts" •  Upgrade Challenges" •  Connection Floods" •  Support to the Rescue" •  Magniﬁed the Reasons to Operationalize"

Slide 5

Slide 5 text

Operation “Operationalize”"

Slide 6

Slide 6 text

Operation “Operationalize”" •  Take Advantage of 10Gen Support" •  Establish Standards" •  Monitoring/Alerting" •  Backups" •  Information Sharing" •  Challenges"

Slide 7

Slide 7 text

10Gen Support" •  JIRA Documents Cases Enabling Information Sharing and Referencing" •  Serves Us Well in Both “Emergencies” and “Hey Quick Question”" •  Training Classes: Administration & Development"

Slide 8

Slide 8 text

Standard Host Setup" •  CentOS 6" •  XFS/EXT4" •  Deadline I/O Scheduler" •  dirty_ratio = 10 (down from 40)" •  dirty_background_ratio = 5 (down from 10)" •  noatime" "

Slide 9

Slide 9 text

Standard Build Scripts" •  Use 64bit Linux Binaries Stored in AOL Repository" •  mongodb_install.sh" – Parameters Include:" •  MongoDB Version" •  Replica Set Name" •  Sharded? (Y/N)" •  Port (27017 is standard)" •  Type (DB, Arbiter, Conﬁg, Mongos)"

Slide 10

Slide 10 text

Standard'Directory/Layouts' •  Based'on'our'standard'MySQL'layout' •  Binaries'in'/opt/local/pkg' •  /db/mongodb/'base'directory' –  Ex.'/db/mongodb/27017' –  Can'Override' •  /db/mongodb/27017Grepset1' •  Under'base'directory' –  data' –  etc'(conﬁg'ﬁles)' –  logs' –  scripts'(ex'stop/start)'

Slide 11

Slide 11 text

ORB" •  AOL Technologies’ Conﬁguration Management Database (CMDB)" •  Integrated with many authoritative data repositories" •  Unique namespace for many operations data points" •  Data model for operations management" •  Projects, Assets, People, Applications, Network Data, HCM" •  SQL Interface"

Slide 12

Slide 12 text

Monitoring/Alerting" " •  Argus" •  Nagios" •  Component Test Tool" •  MMS (MongoDB Monitoring Service)" "

Slide 13

Slide 13 text

Argus" •  Metric and Event" – Collector" – Thresholds " – Management" – Data Viewing" '

Slide 14

Slide 14 text

Argus" •  Capture serverStatus Metrics (once a minute)" •  Capture Replication Lag" •  Great for Trending" •  Great for Capacity Planning" •  Great for Troubleshooting" •  We Also Use for Host Metrics (CPU, I/O, etc)" •  Administrated By a Small and very Busy Group" –  Requested Additions/Changes Can Be Slow"

Slide 15

Slide 15 text

Nagios" •  Great For Fault-detection/Alerting" •  Great For Show Me What Is Currently Broken" •  Great For Service Availability Metrics" •  Flexible" •  Reduces Pressure On NOC" •  Integrates With Netcool, Ignore Tool" •  We Write Our Own Plug-Ins"

Slide 16

Slide 16 text

Component Test Tool" •  Runs Parallel HTTP GETs Across a Group of Hosts" •  Matches Regular Expressions On the Page Content (or HTTP Headers)" •  Displays Number (or Content) of Matches in a HTML Table"

Slide 17

Slide 17 text

MMS"

Slide 18

Slide 18 text

Backups" •  Scripts Delivered With Install" •  Choice Between LVM (default) and Standard (mongodump)" •  Ability to Backup Sharded Setup" '

Slide 19

Slide 19 text

Mongo @ AOL Currently" ' '

Slide 20

Slide 20 text

Mongo @ AOL Currently" •  30+ Active MongoDB Projects" •  500+ Servers in Production" •  Developed in:" –  Java" –  PHP" –  Ruby" –  C#" –  Erlang" –  Scala" –  Perl" –  C++"

Slide 21

Slide 21 text

Information Sharing" •  Internal Wikis/Websites" •  UnUniversity (Organized Brown Bags)" •  Architecture Reviews" •  10Gen JIRA Access'

Slide 22

Slide 22 text

Challenges" •  Developers Love It " •  BUT Requires More Upfront Planning Than They May Realize" •  Should NOT Be Our Default Datastore" •  Better Developer OnBoarding" •  Aggressive Upgrading Is Required" •  Coordination of Driver Upgrades"

Slide 23

Slide 23 text

The Future" •  “MongoDB As a Service”" •  Puppet or Chef" •  Internal Administration Console" – Already has Visual Topologies" •  Internal MongoDB Operations Class" – First One is in July"

Slide 24

Slide 24 text

Resources" •  Recommended Presentation: Todd Dampier’s Rock-Solid Mongo Ops (mongolab)" •  Check out Jon Reed’s MongoDB for Online Advertising at AOL (MongoDB UK 2012)" •  www.meetup.com/Washington-DC- MongoDB-Users-Group/"

Slide 25

Slide 25 text

Thank You!" •  www.slideshare.net/radiocats" •  @radiocats" •  www.linkedin.com/in/mdelnegro" •  Thank you, SeniorGif"