MongoDB on Amazon EC2 - Sandeep Parikh, 10gen

MongoDB Austin 2012

MongoDB and Amazon Web Services are a great match. Learn about how MongoDB's components map to AWS compute and storage instances. In addition, we'll also cover the various deployment scenarios when it comes time to set up MongoDB in production. Throughout the presentation we'll go in-depth and review the specific steps necessary to set up MongoDB in a production deployment and discuss how to backup/restore your data as well.

Transcript

1. Sandeep  Parikh   Technical  Product  Marke4ng   sandeep.parikh@10gen.com   Ge#ng  Started  with  MongoDB   and  Amazon  Web  Services  
2. What  We’ll  Cover   •  MongoDB  components  »  AWS  instances   •  Deployment  configura4ons   •  Storage  configura4on   •  Produc4on  4ps   •  SeFng  up  a  three-­‐node  replica  set  
3. MongoDB  Components   • 64-­‐bit  instances   • Where  your  data  lives   • Needs  RAM  and  Disk  I/O   mongod   • Stateless  request  router   • Lives  on  your  app  server   • Minimal  data  needs   mongos   • Metadata  server  for  sharded  configura4ons   • Minimal  data  needs   • 64-­‐bit  instances   config  
4. Amazon  EC2  Components   •  mongod   –  Standard:  Large  or  Extra  Large   (typical)   –  Hi-­‐Memory:  XL,  XXL  or  4XL   (large  data  sets)   –  Cluster  instances  provide   increased  capacity  and   bandwith   •  mongos   –  Deploy  onto  your  app  server  (32   or  64-­‐bit)   •  config   –  Standard  instances   –  Micro  instance  is  sufficient  
5. DEPLOYMENT  CONFIGURATIONS  
6. Single  Node   •  mongod   –  64-­‐bit  EC2  instance   •  Storage   –  Mul4ple  EBS  volumes   –  RAID  10   –  Configured  using  mdadm   mongod   RAID  10  
7. Replica  Set   mongod   primary   RAID  10   mongod   secondary   RAID  10   mongod   secondary   RAID  10  
8. Replica  Set  –  Using  Zones   mongod   primary   RAID  10   mongod   secondary   RAID  10   mongod   secondary   RAID  10   Zone  3   Zone  2   Zone  1  
9. Replica  Set  –  Mul4ple  Regions   app   Region 1 Region 2 mongod   primary   RAID  10   mongod   secondary   RAID  10   mongod   secondary   RAID  10  
10. Replica  Set  –  Security  Groups   mongod   primary   RAID  10   mongod   secondary   RAID  10   mongod   secondary   RAID  10   app   “application” “database”
11. Sharded  Deployment   Shard  1   Shard  2   Shard  3   config   config   config   app  server   mongos   app  server   mongos   app  server   mongos  
12. CONFIGURING  STORAGE  
13. Recommenda4ons   •  EBS-­‐backed  storage  vs.   instance-­‐based   –  Persistent  vs.  ephemeral   •  RAID  10:  “striped   mirrors”   •  4-­‐8  EBS  volumes  for   best  performance   http://en.wikipedia.org/wiki/Nested_RAID_levels#RAID_1_.2B_0
14. DEPLOYMENT  NOTES  
15. Some  Tips   •  Know  your  deployment   –  Security   –  Instances   –  Storage   •  Configure  filesystem  as  Ext4  or  XFS   •  Reduce  I/O  overhead   –  noa4me,  nodira4me   •  Raise  file  descriptor  limits  
16. More  Tips   •  EBS  snapshots  are  an  easy  way  to  back  up   data   •  Deploy  in  a  trusted  environment   – Consider  authen4ca4on  
17. REPLICA  SET  DEPLOYMENT  STEPS  
18. Security  Groups   •  Create  groups  for  the  following   –  app  servers   –  MongoDB  components   •  Ports  to  remember   –  mongod:  27017,  28017  (web-­‐based  status)   –  mongod  shard  server:  27018     –  mongos:  27017   –  config:  27019   •  Security  groups  act  as  ACLs   •  Instances  are  accessible  via  SSH  keys  
19. Single  Node   •  We’ll  use  this  as  a   star4ng  point   •  Single  EC2  instance   •  4  EBS  volumes   •  RAID10   –  Example:   •  100  GiB  total   •  50  GiB  usable     mongod   RAID  10  
20. Crea4ng  the  Components   $ ec2-run-instances ami-41814f28 -n 1 -g database -k cluster-keypair -t m1.large -z us-east-1a! AMI     count   group   keypair   size   zone   Create an instance Create storage volumes (4x) $ ec2-create-volume –s 25 -z us-east-1a! size   zone  
21. Storage  Configura4on   $ ec2-attach-volume vol-e796108a -i i-11eee072 -d /dev/sdh1! volume   instance   device   Attach storage (4x) Start the RAID $ sudo mdadm --create -l10 -n4 /dev/md0 /dev/sdh*! type   new  device   devices  
22. Storage  Configura4on   $ sudo fdisk /dev/md0! $ sudo mkfs.ext4 /dev/md0p1! Partition device and make filesystem $ sudo mkdir /data! $ sudo chown `id -u` /data! Create mount point and set ownership $ sudo echo ‘/dev/md0p1 /data auto noatime,noexec,nodiratime 0 0’ >> /etc/fstab! $ sudo mount –a /dev/md0p1 /data! Update the filesystem table and mount
23. Install  MongoDB   $ echo "[10gen]! name=10gen Repository! baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64! gpgcheck=0" | sudo tee -a /etc/yum.repos.d/10gen.repo! Update local repo settings $ sudo yum -y install mongo-10gen-server! $ sudo yum -y install sysstat! Install MongoDB and tools $ sudo chkconfig --level 3 mongod off! $ sudo chkconfig --level 5 mongod off! Change auto-start settings
24. MongoDB  Configura4on   $ sudo nano /etc/mongod.conf! ...! dbpath=/data! ...! Set the dbpath $ sudo chkconfig mongod on! $ sudo /etc/init.d/mongod start! Set to autostart at boot and start the server now $ mongo! MongoDB shell version: 2.0.2! connecting to: test! >! Connect to MongoDB
25. Adding  Addi4onal  Nodes   •  For  each  node,  first   repeat  steps  for   –  Crea4ng  instances  and   volumes   –  Configuring  storage   •  Include  replica  set   parameter  in  MongoDB   configura4on   •  Start  mongod  
26. MongoDB  Configura4on   $ sudo nano /etc/mongod.conf! ...! dbpath=/data! replSet=replicaSetName! ...! Set the dbpath $ sudo chkconfig mongod on! $ sudo /etc/init.d/mongod start! Set to autostart at boot and start the server now replica  set  name  
27. Ini4alize  Replica  Set   $ mongo! MongoDB shell version: 2.0.2! connecting to: test! >! Connect to MongoDB > rs.initiate()! {! "info2" : "no configuration explicitly specified -- making one",! "me" : "ip-10-127-127-91:27017",! "info" : "Config now saved locally. Should come online in about a minute.",! "ok" : 1! }! Initialize the replica set > rs.add(“ec2-abc.def.amazonaws.com”)! { “ok” : 1 }! Add each replica set member external  DNS  name  
28. Ini4alize  Replica  Set   >! The mongo prompt should go from this PRIMARY>! To this SECONDARY>! Or this
29. BACKUP/RESTORE  
30. Backing  Up  Your  Data   $ mongo! SECONDARY> use admin! SECONDARY> db.runCommand({fsync:1,lock:1});! {! "info" : "now locked against writes, use db.$cmd.sys.unlock.findOne() to unlock",! "ok" : 1! }! Lock the database $ ec2-create-snapshot vol-1234abcd --description "MongoDB RAID backup1"! volume   descrip4on   Create a snapshot for each attached volume $ mdadm --detail /dev/md0p1! Run mdadm and note the UUID
31. Backing  Up  Your  Data   $ mongo! SECONDARY> db.$cmd.sys.unlock.findOne();! { "ok" : 1, "info" : "unlock requested" }! Unlock the database
32. Restoring  Your  Data   $ ec2-create-volume --availability-zone us-east-1a --snapshot vol-1234abcd! $ ec2-attach-volume vol-1234abcd -i i-aa3bc4c8 -d /dev/sdh1! Create and attach a volume from each snapshot $ sudo mdadm --assemble --auto-update-homehost -u 07552c4d…a9c2f2fc --no-degraded /dev/md0! mdadm: /dev/md0 has been started with 4 drives.! Reassemble the RAID $ sudo mkdir /data2! $ sudo chown `id -u` /data2! Create mount point and set ownership UUID  from  mdadm  
33. Restoring  Your  Data   Update the filesystem table and mount $ sudo echo ‘/dev/md0p1 /data2 auto noatime,noexec,nodiratime 0 0’ >> /etc/fstab! $ sudo mount –a /dev/md0p1 /data2! $ mongod --dbpath /data2! $ mongo! MongoDB shell version: 2.0.2! connecting to: test! > use yourDb! > db.yourCollection.validate({full:true})! Verify the restore (optional, requires time and disk space) $ sudo cp /data2 /data! Set /data2 to /data
34. Recommended  Backup  Methods   •  Create  backups  on  secondary  servers  when   using  replica4on   – Doesn’t  block  on  the  primary   – Secondary  updates  once  unlocked   •  Use  mongodump  to  backup  live  data   – Collec4ons,  databases,  everything   – Restore  with  mongorestore  
35. QUESTIONS?  
36. Learn  More   •  MongoDB  on  Amazon  Web  Services  white  paper:   hnp://www.10gen.com/white-­‐papers   •  Amazon  EC2  info  at  MongoDB  wiki:   hnp://www.mongodb.org/display/DOCS/Amazon+EC2   •  Scripts  for  working  with  EBS  (crea4ng,  anaching,  etc.):   hnps://gist.github.com/1482182      
37. Contact  Info   •  MongoDB   –  hnp://www.mongodb.org   –  Downloads,  Docs,  Forums,  etc.   •  10gen   –  hnp://www.10gen.com/contact   –  info@10gen.com   –  Consul4ng,  Support,  etc.   •  Sandeep  Parikh   –  sandeep.parikh@10gen.com