Deploying MongoDB on AWS using CloudFormation

CloudFormation gives developers and system administrators an easy way to describe and provision collections of AWS resources. In this webinar, we'll show you how to use Cloud Formation to create templates for your own deployments including coordination of multi-node clusters and RAID10 enabled EBS volumes. We'll share some example code, best practices for MongoDB on AWS, and walk you through the process of creating your own templates.

Transcript

1. Sandeep  Parikh   Technical  Product  Manager   sandeep.parikh@10gen.com   @crcsmnky   Automa'ng  Deployment  with   CloudForma'on   Thanks for joining, we’ll begin shortly
2. Agenda   •  MongoDB  basics   – Components   – Deployment  types   – Storage   •  CloudFormaBon   – Sample  template  background   – Template  walkthrough   •  Single  Node   •  Replica  Set  
3. MongoDB  Components   mongod   Core   database   server   process   mongos   Request   router  for   sharded   deployments   config   Metadata   server  for   sharded   deployments  
4. Amazon  EC2  Components   •  mongod  (64-­‐bit)   –  Standard:  L,  XL     –  Hi-­‐Memory:  XL,  XXL  or  4XL   –  Cluster:  increased  capacity   and  bandwidth   •  mongos   –  Deploy  onto  your  app  server   •  config   –  Standard  instances  sufficient  
5. DEPLOYMENT  CONFIGURATIONS  
6. Single  Node   mongod   RAID  10  
7. Replica  Set   mongod   primary   RAID  10   mongod   secondary   RAID  10   mongod   secondary   RAID  10  
8. Replica  Set  –  MulBple  Zones   app   Zone 1 Zone 3 mongod   primary   RAID  10   mongod   secondary   RAID  10   mongod   secondary   RAID  10   Zone 2
9. Replica  Set  –  MulBple  Regions   app   Region 1 Region 2 mongod   primary   RAID  10   mongod   secondary   RAID  10   mongod   secondary   RAID  10  
10. Replica  Set  –  Security  Groups   mongod   primary   RAID  10   mongod   secondary   RAID  10   mongod   secondary   RAID  10   app   “application” “database”
11. Sharded  Deployment   Shard  1   Shard  2   Shard  3   config   config   config   app  server   mongos   app  server   mongos   app  server   mongos  
12. CONFIGURING  DEPLOYMENT  
13. Storage  ConsideraBons   •  EBS-­‐backed  storage  vs.   instance-­‐based   –  Persistent  vs.  ephemeral   •  RAID  10:  “striped   mirrors”   •  4-­‐8  EBS  volumes  for   best  performance   http://en.wikipedia.org/wiki/Nested_RAID_levels#RAID_1_.2B_0
14. Deployment  Tips   •  Know  your  deployment   –  Security  groups   –  Instance  sizes   –  Storage  needed  now  and  later   •  Configure  filesystem  as  ext4  or  XFS   •  Reduce  I/O  overhead   •  Raise  file  descriptor  limits   •  Set  read-­‐ahead  
15. CLOUDFORMATION  
16. AWS  CloudFormaBon   •  Gives  devs  and  admins  easy  way  to  create   and  manage  AWS  resources   •  Create  text-­‐based  templates  which  describe   resources  in  your  stack   •  Templates  are  JSON  and  can  be  managed  like   code  or  stored  in  S3  
17. MongoDB  plus  CloudFormaBon   •  Reference  templates  for  building  your  own   deployment   – Single  node  for  test  and  development   – 3-­‐node  replica  set  for  producBon-­‐level   deployments   •  Customize  them  for  your  own  needs  
18. Sample  Templates   •  Single-­‐node   – Single  mongod  with  4  EBS  volumes  in  a  RAID10   •  Replica  set   – Each  node  similar  to  single  node   – Primary  configured  in  “parent”  template   – Secondary  configured  in  “child”  template   – MulBple  secondaries  are  created/referenced  in   “parent”  template  
19. Resources  Created   •  Based  on  Amazon  Linux  (for  your  region)   •  New  security  group   – Port  22:  open  to  all   – Port  27017:  open  to  instances  in  group   •  IAM  resource   •  4  EBS  volumes   •  4  EBS  volume  agachments   •  10-­‐minute  Bmeout  for  the  whole  thing  
20. Inputs  
21. Mappings  
22. Security  Groups  
23. Security  Groups  –  Gotchas   •  Don’t  forget:   – App  servers  need  to  talk  to  MongoDB   – Sharding  uses  addiBonal  ports  (27018,  27019)   – Status  page  is  visible  
24. Storage  Resources   x 4 x 4
25. Instance  ConfiguraBon     •  Init  instance   –  Install  packages   –  Add  10gen  repo   •  Wait  for  EBS  volumes  to  agach   •  Install  MongoDB   •  Storage  setup   –  Create  RAID10   –  Blockdev  read-­‐ahead  for  each  device   –  LVM  setup   •  MongoDB  configuraBon   •  Finish  
26. Instance  Metadata  
27. Instance  ProperBes   •  Based  on  the  region  you’re  running  in  and   instance  size  input,  choose  an  AMI   •  KeyName  is  an  input  parameter   •  UserData  holds  configuraBon  script  
28. Script  –  Gelng  Started   •  Install  AWS  CloudFormaBon  tools   •  Error  helper  funcBon   •  Access  and  Secret  Keys  from  IAM  resource  
29. Script  –  Disks,  MongoDB,  RAID   x 4 x 4
30. Script  –  LVM,  FS,  Mounts   x 3 x 3 •  Data,  Log  and  Journal  each  have     –  their  own  volume   –  Mount  path/pount  
31. Script  –  Configure,  Start  MongoDB   •  Signal  clears  WaitCondiBon  (10  minute   Bmeout)  with  success  
32. Replica  Set  Template   ReplicaSetStack   ReplicaSetMember   ReplicaSetMember  
33. Replica  Set  Member   •  Child  templates  must  be  in  S3   •  Note:  these  are  set  to  be  in  a  different  AZ  
34. Replica  Set  ConfiguraBon   •  Almost  the  same  as  single  node   •  Added  “replSet”  parameter  to  mongod.conf   •  All  Replica  Set  nodes  have  this  
35. Replica  Set  IniBalizaBon   •  Created  Replica  Set  init  script   •  $HOSTNAME  pulled  from  EC2  instance  metadata   •  Run  on  the  instance  from  ReplicaSetStack  
36. Adding  More  Storage   •  Add  Volume  resource   •  Add  VolumeAgachment  resource   •  Add  sleep  condiBon  in  script   •  Edit  mdadm  for  addiBonal  volumes  in  script  
37. Adding  More  Instances   •  Edit  the  ReplicaSetStack  template   •  Add  addiBonal  “ReplicaSetMember”  instances   •  Update  “replicaSetConfigInit.js”   •  Spread  across  mulBple  zones   – Note:  spreading  across  regions  may  require   addiBonal  edits  and/or  manual  inputs  
38. Using  Templates   •  Go  to  AWS  Management  Console  >  AWS   CloudFormaBon  >  Create  New  Stack   •  ReplicaSetMember  template  must  be  in  S3   •  ReplicaSetStack  template  needs  that  S3  URL   – See  “TemplateURL”  
39. Download  Sample  Templates   •  hgp://www.mongodb.org/display/DOCS/ AutomaBng+Deployment+with+Cloud +FormaBon   – MongoDB_SingleNode.template   – MongoDB_ReplicaSetStack.template   – MongoDB_ReplicaSetMember.template  
40. WRAP  UP  
41. Learn  More   •  AutomaBng  Deployment  on  AWS  with  CloudFormaBon   hgp://www.mongodb.org/display/DOCS/AutomaBng +Deployment+with+CloudFormaBon   •  Amazon  EC2  info  (incl.  in-­‐depth  guide):   hgp://www.mongodb.org/display/DOCS/Amazon+EC2   •  MongoDB  AMI  on  AWS  Marketplace   hgps://aws.amazon.com/marketplace/pp/B007IBMJPI/   •  MongoDB  on  Amazon  Web  Services  white  paper:   hgp://www.10gen.com/white-­‐papers  
42. QuesBons  &  Comments   •  Sandeep  Parikh   –  sandeep.parikh@10gen.com   –  @crcsmnky   •  MongoDB   –  hgp://www.mongodb.org   –  Downloads,  Docs,  Forums,  etc.   •  10gen   –  hgp://www.10gen.com/contact   –  info@10gen.com   –  ConsulBng,  Support,  etc.