MongoDB and Amazon Web Services - Sandeep Parikh

Webinar 1/20/2012

MongoDB and Amazon Web Services are a great match. Learn about how MongoDB's components map to AWS compute and storage instances. In addition, we'll also cover the various deployment scenarios when it comes time to set up MongoDB in production. Throughout the presentation we'll go in-depth and review the specific steps necessary to set up MongoDB in a production deployment and discuss how to backup/restore your data as well.

Transcript

1. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues Thanks  for  joining     “MongoDB  and  Amazon  Web  Services”   We’ll  begin  shortly      
2. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues About  10gen   •  10gen  develops  MongoDB,  and  offers  producGon   support,  training,  and  consulGng  for  the  open  source   database.    
3. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues What  We’ll  Cover   •  MongoDB  components  »  AWS  instances   •  Deployment  configuraGons   •  Storage  configuraGon   •  ProducGon  Gps   •  Deploying  a  three-­‐node  replica  set  
4. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues MongoDB  Components   • 64-­‐bit  instances   • Where  your  data  lives   • Needs  RAM  and  Disk  I/O   mongod   • Stateless  request  router   • Lives  on  your  app  server   • Minimal  data  needs   mongos   • Metadata  server  for  sharded  configuraGons   • Minimal  data  needs   • 64-­‐bit  instances   config  
5. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues Amazon  EC2  Components   •  mongod   –  Standard:  Large  or  Extra  Large   (typical)   –  Hi-­‐Memory:  XL,  XXL  or  4XL   (large  data  sets)   –  Cluster  instances  provide   increased  capacity  and   bandwith   •  mongos   –  Deploy  onto  your  app  server  (32   or  64-­‐bit)   •  config   –  Standard  instances   –  Micro  instance  is  sufficient  
6. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues DEPLOYMENT  CONFIGURATIONS  
7. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues Single  Node   •  mongod   –  64-­‐bit  EC2  instance   •  Storage   –  MulGple  EBS  volumes   –  RAID  10   –  Configured  using  mdadm   mongod   RAID  10  
8. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues Replica  Set   mongod   primary   RAID  10   mongod   secondary   RAID  10   mongod   secondary   RAID  10  
9. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues Replica  Set  –  Using  Zones   mongod   primary   RAID  10   mongod   secondary   RAID  10   mongod   secondary   RAID  10   Zone  3   Zone  2   Zone  1  
10. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues Replica  Set  –  MulGple  Regions   app   Region 1 Region 2 mongod   primary   RAID  10   mongod   secondary   RAID  10   mongod   secondary   RAID  10  
11. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues Replica  Set  –  Security  Groups   mongod   primary   RAID  10   mongod   secondary   RAID  10   mongod   secondary   RAID  10   app   “application” “database”
12. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues Sharded  Deployment   Shard  1   Shard  2   Shard  3   app  server   config   config   config   app  server   app  server   app  server  
13. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues CONFIGURING  STORAGE  
14. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues RecommendaGons   •  EBS-­‐backed  storage  vs.   instance-­‐based   –  Persistent  vs.  ephemeral   •  RAID  10:  “striped   mirrors”   •  4-­‐8  EBS  volumes  for   best  performance   http://en.wikipedia.org/wiki/Nested_RAID_levels#RAID_1_.2B_0
15. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues DEPLOYMENT  NOTES  
16. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues Some  Tips   •  Know  your  deployment   –  Security   –  Instances   –  Storage   •  Configure  filesystem  as  Ext4  or  XFS   •  Reduce  I/O  overhead   –  noaGme,  nodiraGme   •  Raise  file  descriptor  limits  
17. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues More  Tips   •  EBS  snapshots  are  an  easy  way  to  back  up   data   •  Deploy  in  a  trusted  environment   – Consider  authenGcaGon  
18. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues REPLICA  SET  DEPLOYMENT  STEPS  
19. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues Security  Groups   •  Create  groups  for  the  following   –  app  servers   –  MongoDB  components   •  Ports  to  remember   –  mongod:  27017,  28017  (web-­‐based  status)   –  mongod  shard  server:  27018     –  mongos:  27017   –  config:  27019   •  Security  groups  act  as  ACLs   •  Instances  are  accessible  via  SSH  keys  
20. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues Single  Node   •  We’ll  use  this  as  a   starGng  point   •  Single  EC2  instance   •  4  EBS  volumes   •  RAID10   –  Example:   •  100  GiB  total   •  50  GiB  usable     mongod   RAID  10  
21. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues CreaGng  the  Components   $ ec2-run-instances ami-41814f28 -n 1 -g database -k cluster-keypair -t m1.large -z us-east-1a! AMI     count   group   keypair   size   zone   Create an instance Create storage volumes (4x) $ ec2-create-volume –s 25 -z us-east-1a! size   zone  
22. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues Storage  ConfiguraGon   $ ec2-attach-volume vol-e796108a -i i-11eee072 -d /dev/sdh1! volume   instance   device   Attach storage (4x) Start the RAID $ sudo mdadm --create -l10 -n4 /dev/md0 /dev/sdh*! type   new  device   devices  
23. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues Storage  ConfiguraGon   $ sudo fdisk /dev/md0! $ sudo mkfs.ext4 /dev/md0p1! Partition device and make filesystem $ sudo mkdir /data! $ sudo chown `id -u` /data! Create mount point and set ownership $ sudo echo ‘/dev/md0p1 /data auto noatime,noexec,nodiratime 0 0’ >> /etc/fstab! $ sudo mount –a /dev/md0p1 /data! Update the filesystem table and mount
24. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues Install  MongoDB   $ echo "[10gen]! name=10gen Repository! baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64! gpgcheck=0" | sudo tee -a /etc/yum.repos.d/10gen.repo! Update local repo settings $ sudo yum -y install mongo-10gen-server! $ sudo yum -y install sysstat! Install MongoDB and tools $ sudo chkconfig --level 3 mongod off! $ sudo chkconfig --level 5 mongod off! Change auto-start settings
25. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues MongoDB  ConfiguraGon   $ sudo nano /etc/mongod.conf! ...! dbpath=/data! ...! Set the dbpath $ sudo chkconfig mongod on! $ sudo /etc/init.d/mongod start! Set to autostart at boot and start the server now $ mongo! MongoDB shell version: 2.0.2! connecting to: test! >! Connect to MongoDB
26. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues Adding  AddiGonal  Nodes   •  For  each  node,  first   repeat  steps  for   –  CreaGng  instances  and   volumes   –  Configuring  storage   •  Include  replica  set   parameter  in  MongoDB   configuraGon   •  Start  mongod  
27. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues MongoDB  ConfiguraGon   $ sudo nano /etc/mongod.conf! ...! dbpath=/data! replSet=replicaSetName! ...! Set the dbpath $ sudo chkconfig mongod on! $ sudo /etc/init.d/mongod start! Set to autostart at boot and start the server now replica  set  name  
28. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues IniGalize  Replica  Set   $ mongo! MongoDB shell version: 2.0.2! connecting to: test! >! Connect to MongoDB > rs.initiate()! {! "info2" : "no configuration explicitly specified -- making one",! "me" : "ip-10-127-127-91:27017",! "info" : "Config now saved locally. Should come online in about a minute.",! "ok" : 1! }! Initialize the replica set > rs.add(“ec2-abc.def.amazonaws.com”)! { “ok” : 1 }! Add each replica set member external  DNS  name  
29. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues IniGalize  Replica  Set   >! The mongo prompt should go from this PRIMARY>! To this SECONDARY>! Or this
30. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues BACKUP/RESTORE  
31. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues Backing  Up  Your  Data   $ mongo! SECONDARY> use admin! SECONDARY> db.runCommand({fsync:1,lock:1});! {! "info" : "now locked against writes, use db.$cmd.sys.unlock.findOne() to unlock",! "ok" : 1! }! Lock the database $ ec2-create-snapshot vol-1234abcd --description "MongoDB RAID backup1"! volume   descripGon   Create a snapshot for each attached volume $ mdadm --detail /dev/md0p1! Run mdadm and note the UUID
32. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues Backing  Up  Your  Data   $ mongo! SECONDARY> db.$cmd.sys.unlock.findOne();! { "ok" : 1, "info" : "unlock requested" }! Unlock the database
33. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues Restoring  Your  Data   $ ec2-create-volume --availability-zone us-east-1a --snapshot vol-1234abcd! $ ec2-attach-volume vol-1234abcd -i i-aa3bc4c8 -d /dev/sdh1! Create and attach a volume from each snapshot $ sudo mdadm --assemble --auto-update-homehost -u 07552c4d…a9c2f2fc --no-degraded /dev/md0! mdadm: /dev/md0 has been started with 4 drives.! Reassemble the RAID $ sudo mkdir /data2! $ sudo chown `id -u` /data2! Create mount point and set ownership UUID  from  mdadm  
34. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues Restoring  Your  Data   Update the filesystem table and mount $ sudo echo ‘/dev/md0p1 /data2 auto noatime,noexec,nodiratime 0 0’ >> /etc/fstab! $ sudo mount –a /dev/md0p1 /data2! $ mongod --dbpath /data2! $ mongo! MongoDB shell version: 2.0.2! connecting to: test! > use yourDb! > db.yourCollection.validate({full:true})! Verify the restore (optional, requires time and disk space) $ sudo cp /data2 /data! Set /data2 to /data
35. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues Recommended  Backup  Methods   •  Create  backups  on  secondary  servers  when   using  replicaGon   – Doesn’t  block  on  the  primary   – Secondary  updates  once  unlocked   •  Use  mongodump  to  backup  live  data   – CollecGons,  databases,  everything   – Restore  with  mongorestore  
36. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues QUESTIONS?  
37. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues Learn  More   •  MongoDB  on  Amazon  Web  Services  white  paper:   hnp://www.10gen.com/white-­‐papers   •  Amazon  EC2  info  at  MongoDB  wiki:   hnp://www.mongodb.org/display/DOCS/Amazon+EC2   •  Scripts  for  working  with  EBS  (creaGng,  anaching,  etc.):   hnps://gist.github.com/1482182   •  The  slides  and  audio  recording  will  be  available  at:   hnp://www.10gen.com/events    
38. To join via VOIP, go to Communicate > Integrated Voice Conference > Join Conference Contact webinars@10gen.com if you are having audio issues Contact  Info   •  MongoDB   –  hnp://www.mongodb.org   –  Downloads,  Docs,  Forums,  etc.   •  10gen   –  hnp://www.10gen.com/contact   –  info@10gen.com   –  ConsulGng,  Support,  etc.   •  Sandeep  Parikh     –  @crcsmnky   –  Background  in  soqware  engineering,  developed  soqware  to  compute   metrics  across  large  graphs,  analyze  social  networks  and  measure  text   similarity.