Share
Embed
MongoDB 2.8 and MongoDB on AWS
A few notes about running MongoDB on AWS
Randall Hunt
December 05, 2014
Transcript
36 slides
-
© 2014 Amazon.com, Inc. and its affiliates. All rights reserved.
May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc. December 05, 2014 | SFO AWS Pop Up Loft MongoDB on AWS J. Randall Hunt, AWS @jrhunt -
Assumptions • You're familiar with MongoDB • You've run production
workloads with MongoDB • You're familiar with NoSQL vocabulary • If you have questions ask them now. -
Disclaimer • I don't work for MongoDB • Some of
this could be wrong • Run all benchmarks yourself! • Test your workloads now -
MongoDB 2.8 Major Changes • Pluggable Storage Engines • Collection
Level Locking • Larger Replica Set Support • Better Authentication Methods • Better Query Introspection • Toolchain rewritten in Go -
Pluggable Storage Engine • Wired Tiger: – Storage engine written
for modern hardware – Focuses on efficient use of RAM • MMAPv1 – Default, traditional – Collection Level Locking – What you're used to -
Wired Tiger • Mixed storage formats (B-tree, column, LSM) •
non-blocking access and write implementations • no in-pace updates • use cache resources • compression choices -
Wired Tiger Compression, zlib storage: dbPath: "/ssd/db/wt_zlib" engine: "wiredTiger" wiredTiger:
collectionConfig: "block_compressor=zlib" -
Larger Replica Set Support • Replicasets can now have 50
members • Still only 7 voting members • Replica set code rewritten, many more tests • More stable (in my tests) -
SCRAM-SHA1 Authentication • Salted Challenge Response Authentication Mechanism • SASL
mechanism • RFC 5802 • Even if data is taken damage is limited -
Query Introspection • .explain() works on every operation • .explain()
provides detailed information about query planner -
Entire Toolchain Re-written In Go • https://github.com/mongodb/mongo-tools/ • bsondump (explore
bson files) • mongoimport/export (csv/json import/export) • mongodump/restore (bson import/export) • mongostat (monitor servers or clusters) • mongofiles (IO for gridfs) • mongooplog (replay oplog entries) • mongotop (like top for mongod) -
CloudFormation Templates • https://github.com/crcsmnky/aws-cfn-mongodb • Not terribly up to date
• Doesn't work in every region • Infrastructure as code! • Easy to fork and edit • Good bootstrapping for Chef, Puppet, etc. -
Runtime Options • Deploy on generic EC2 • Deploy as
an opsworks stack • Deploy with elastic beanstalk (so cool) • Deploy with EMR integration -
Elastic Beanstalk • Autoscale App Servers • Autoscale MongoS •
Manage application/drivers in one place -
Elastic Load Balancer App Server mongos App Server mongos App
Server mongos MongoDB Cluster Security Group Auto Scaling Group -
Elastic Map Reduce • https://github.com/mongodb/mongo-hadoop • MongoDB-Hadoop bi-directional connector •
Hive and Pig integrations • Works with BSON or live MongoDB -
Infrastructure • EIPs • VPC • IAM • Route53 (DNS)
• CloudWatch -
EC2 Instance Types • General Purpose (t2,m3) • Compute Optimized
(c3) • GPU (G2) • Memory Optimized (R3) • Storage Optimized (I2,HS1) -
Sizing Details mongod core process memory, network, storage m3, i2,
r3 config server sharding metadata low power t2, m3 mongos query router low power deploy with app servers -
Questions To Ask For Production • How much availability do
I really need? • How much IO do I really need? • Can I withstand the loss of a zone? of a region? • Where are my users coming from? • What are my security requirements? -
Storage Options • EBS – GP2 (fast, burstable, credit system)
– PIOPS (stable) – Magnetic • Instance – Ephemeral (fast but scary) -
Instance Configuration • Use Amazon Linux on HVM • Install
from the mongodb repo with yum • Use PIOPS EBS for consistent performance • Use EBS optimized -
Storage Settings (more important) • Separate EBS volumes for data,
log, journal • Use EXT4 or XFS (but XFS can be bad) • Set read ahead to 16k (blockdev --setra) (for EBS volumes only) • No need for RAID 10, just use RAID 0 • Mount noatime, noexec, nodiratime -
Networking Settings • set ulimits to max • set TCP
keepalive to very high • use CNAMEs for servers and configuration so that it's easy to upgrade/add/remove instnaces • Use an IAM mongodb role -
Metrics (Cloudwatch + Alerting) • Storage: – Queue depth –
Read/Write latency -
Tips and Tricks • Use a delayed secondary • dump/restore
won't work in a timely manner for larger deployments, use MMS backup or EBS snapshots instead: https://github.com/msaffitz/ mongolly • Use MMS for everything. -
Please give us your feedback on this presentation © 2014
Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc. #awspopuploft