MongoDB and Amazon Web Services - MongoSF 2012

1. Sandeep  Parikh   Technical  Product  Manager   sandeep.parikh@10gen.com   @crcsmnky   MongoDB     and     Amazon  Web  Services  
2. Agenda   •  MongoDB  basics   – Components   – Deployment  types   – Storage   •  Deploying  a  Replica  Set   •  AWS  Marketplace   •  CloudFormaDon  
3. MongoDB  Components   mongod   Core   database   server   process   mongos   Request   router  for   sharded   deployments   config   Metadata   server  for   sharded   deployments  
4. Amazon  EC2  Components   •  mongod   –  Standard:  L,  XL     –  Hi-­‐Memory:  XL,  XXL  or  4XL   –  Cluster:  increased  capacity   and  bandwidth   •  mongos   –  Deploy  onto  your  app  server   •  config   –  Standard  instances  sufficient  
5. DEPLOYMENT  CONFIGURATIONS  
6. Single  Node   mongod   RAID  10  
7. Replica  Set   mongod   primary   RAID  10   mongod   secondary   RAID  10   mongod   secondary   RAID  10  
8. Replica  Set  –  MulDple  Zones   app   Zone 1 Zone 3 mongod   primary   RAID  10   mongod   secondary   RAID  10   mongod   secondary   RAID  10   Zone 2
9. Replica  Set  –  MulDple  Regions   app   Region 1 Region 2 mongod   primary   RAID  10   mongod   secondary   RAID  10   mongod   secondary   RAID  10  
10. Replica  Set  –  Security  Groups   mongod   primary   RAID  10   mongod   secondary   RAID  10   mongod   secondary   RAID  10   app   “application” “database”
11. Sharded  Deployment   Shard  1   Shard  2   Shard  3   config   config   config   app  server   mongos   app  server   mongos   app  server   mongos  
12. CONFIGURING  DEPLOYMENT  
13. Storage  ConsideraDons   •  Instance  vs.  EBS   –  Ephemeral  vs.  persistent   •  RAID  10   –  “striped  mirrors”   •  4-­‐8  EBS  volumes  for   best  performance   http://en.wikipedia.org/wiki/Nested_RAID_levels#RAID_1_.2B_0
14. Deployment  Tips   •  Know  your  deployment   –  Security  groups   –  Instance  sizes   –  Storage  needed  now  and  later   •  Configure  filesystem  as  ext4  or  XFS   •  Reduce  I/O  overhead   •  Raise  file  descriptor  limits   •  Set  read-­‐ahead  
15. REPLICA  SET  DEPLOYMENT  STEPS  
16. Single  Node   •  We’ll  use  this  as  a   starDng  point   •  Single  EC2  instance   •  4  EBS  volumes   •  RAID10   –  Example:   •  100  GiB  total   •  50  GiB  usable     mongod   RAID  10  
17. CreaDng  the  Components   $ ec2-run-instances ami-41814f28 -n 1 -g database -k cluster-keypair -t m1.large -z us-east-1a! AMI     count   group   keypair   size   zone   Create an instance Create storage volumes (4x) $ ec2-create-volume –s 25 -z us-east-1a! size   zone  
18. Storage  ConfiguraDon   $ ec2-attach-volume vol-e796108a -i i-11eee072 -d /dev/sdh1! volume   instance   device   Attach storage (4x) Start the RAID $ sudo mdadm --create --level=l10 --raid-devices=4 /dev/md0 /dev/sdh*! type   new  device   devices  
19. Storage  ConfiguraDon   $ sudo pvcreate /dev/md0! $ sudo vgcreate vg0 /dev/md0! $ sudo lvcreate -n data vg0! $ sudo mkfs.ext4 /dev/vg0/data! Create volumes and filesystem $ sudo mkdir /data! $ sudo chown mongod:mongod /data! Create mount point and set ownership $ echo ‘/dev/vg0/data /data ext4 defaults,auto,noatime,noexec 0 0’ | sudo tee –a /etc/fstab! $ sudo mount /data! Update the filesystem table and mount
20. Install  MongoDB   $ echo "[10gen]! name=10gen Repository! baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64! gpgcheck=0" | sudo tee -a /etc/yum.repos.d/10gen.repo! Update local repo settings $ sudo yum -y install mongo-10gen-server! $ sudo yum -y install sysstat! Install MongoDB and tools
21. MongoDB  ConfiguraDon   $ sudo $EDITOR /etc/mongod.conf! ...! dbpath=/data! ...! Set the dbpath $ sudo /etc/init.d/mongod start! Start the server now $ mongo! MongoDB shell version: 2.0.4! connecting to: test! >! Connect to MongoDB
22. Adding  AddiDonal  Nodes   •  For  each  node,  first   repeat  steps  for   –  CreaDng  instances  and   volumes   –  Configuring  storage   •  Include  replica  set   parameter  in  MongoDB   configuraDon   •  Start  mongod  
23. MongoDB  ConfiguraDon   $ sudo $EDITOR /etc/mongod.conf! ...! dbpath=/data! replSet=replicaSetName! ...! Set the dbpath $ sudo /etc/init.d/mongod start! Start the server now replica  set  name  
24. IniDalize  Replica  Set   $ mongo! MongoDB shell version: 2.0.4! connecting to: test! >! Connect to MongoDB > rs.initiate()! {! "info2" : "no configuration explicitly specified -- making one",! "me" : "ip-10-127-127-91:27017",! "info" : "Config now saved locally. Should come online in about a minute.",! "ok" : 1! }! Initialize the replica set > rs.add(“ec2-abc.def.amazonaws.com”)! { “ok” : 1 }! Add each replica set member external  DNS  name  
25. Replica  Set  Shell  Prompt   >! The mongo prompt should go from this PRIMARY>! To this SECONDARY>! Or this
26. MARKETPLACE  
27. AWS  Marketplace   •  A  place  to  browse  and  deploy  pre-­‐built  AMIs   •  Free  and  paid  opDons   •  One-­‐click  deploy  (well,  maybe  three)   •  Sofware  available  from  a  variety  of  vendors   •  Deployable  onto  almost  any  instance  
28. AWS  Marketplace  
29. MongoDB  AMI   •  10gen  created   •  Amazon  Linux   •  MongoDB  pre-­‐ installed   •  InstrucDons  for   storage  and   MongoDB   configuraDon  
30. CLOUDFORMATION  
31. AWS  CloudFormaDon   •  Gives  devs  and  admins  easy  way  to  create   and  manage  AWS  resources   •  Create  text-­‐based  templates  which  describe   resources  in  your  stack   – EC2,  EBS,  Load  Balancer,  CDN   •  Templates  are  JSON   – managed  like  code  and  pushed  to  S3  
32. MongoDB  via  CloudFormaDon   •  Reference  templates  for  building  your  own   deployment   – Single  node  for  test  and  development   – 3-­‐node  replica  set  for  producDon-­‐level   deployments   •  Customize  them  for  your  own  needs  
33. MongoDB  Templates   •  Single-­‐node   – Single  mongod  with  4  EBS  volumes  in  a  RAID10   •  Replica  set   – Each  node  similar  to  single  node   – Primary  configured  in  “parent”  template   – Secondary  configured  in  “child”  template   – “Parent”  template  refers  to  “child”  template  to   create  2  secondary  instances  
34. Template  ConfiguraDon   •  Amazon  Linux   •  Inputs   – Key  Name,  Instance  Size,  Volume  Size   •  New  security  group   – Port  22:  open  to  all   – Port  27017:  open  to  instances  in  group   •  4  EBS  volumes,  ajachments    
35. Template  Inputs  
36. Template  ConfiguraDon   •  Instance  installs  MongoDB  via  yum   •  Creates  RAID10   •  Configures  storage  (LVM,  FS,  etc.)   •  Configures  MongoDB  
37. Customizing  Templates   •  Add  more  storage  volumes   •  Add  more  instances   •  Customize  security  group  rules   •  Customize  zones/regions  
38. Download  Sample  Templates   •  hjp://www.mongodb.org/display/DOCS/ AutomaDng+Deployment+with +CloudFormaDon   – MongoDB_SingleNode.template   – MongoDB_ReplicaSetStack.template   – MongoDB_ReplicaSetMember.template  
39. WRAP  UP  
40. Learn  More   •  Amazon  EC2  info  (incl.  in-­‐depth  guide):   hjp://www.mongodb.org/display/DOCS/Amazon+EC2     •  AutomaDng  Deployment  on  AWS  with  CloudFormaDon   hjp://www.mongodb.org/display/DOCS/AutomaDng +Deployment+with+CloudFormaDon     •  MongoDB  AMI  via  AWS  Marketplace   hjps://aws.amazon.com/marketplace/pp/B007IBMJPI/     •  MongoDB  on  Amazon  Web  Services  white  paper:   hjp://www.10gen.com/white-­‐papers  
41. QuesDons  &  Comments   •  Sandeep  Parikh   –  sandeep.parikh@10gen.com   –  @crcsmnky   •  MongoDB   –  hjp://www.mongodb.org   –  Downloads,  Docs,  Forums,  etc.   •  10gen   –  hjp://www.10gen.com/contact   –  info@10gen.com   –  Training,  ConsulDng,  Support