Chef to scale MongoDB Clusters

The cloud presents many opportunities to use APIs for acquiring virtual hardware resources, servers, storage volumes and networking. This talk will show how to create a very big cluster of MongoDB sharded replica sets with the many considerations required for its operation coded using Chef. The talk will go over using Chef to provision AWS resources, managing secrets and shared data using data bags and vault, and the operations aspects of managing a large MongoDB cluster. These are lessons we learned creating such a cluster for a client who wanted to move from using a MongoDB as a service to their own servers in AWS.

http://www.meetup.com/Chef-Users-London/events/229071046/

Transcript

1. .co.uk www. MongoDB & Chef Operations Automation Right?
2. .co.uk www. Evgeny Zislis Co-Founder / CTO www.devops.co.il www.devopspro.co.uk
3. .co.uk www.
4. .co.uk www.
5. .co.uk www. mongodb-is-web-scale.com
6. .co.uk www.
7. .co.uk www.
8. .co.uk www.
9. .co.uk www. github.com/chef-brigade/mongodb-cookbook github.com/sunggun-yu/chef-mongodb3
10. .co.uk www. ### mongodb/metadata.rb depends 'apt' ### mongodb/recipes/repo.rb include_recipe 'apt' apt_repository 'mongodb-org' do uri 'http://repo.mongodb.org/apt/ubuntu' distribution "#{node['lsb']['codename']}/mongodb-org/3.2" components [ 'multiverse' ] keyserver 'keyserver.ubuntu.com' key 'EA312927' action :add end docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/ github.com/sunggun-yu/chef-mongodb3/blob/master/recipes/package_repo.rb
11. .co.uk www. ### mongodb / attributes / default.rb include_attribute 'sysctl' override['sysctl']['params'] = { # more ports for open connections - 37k # 27017 is MongoDB listen port! 'net.ipv4.ip_local_port_range' => '28000 65000', # more concurrent connections when spammed 'net.ipv4.tcp_fin_timeout' => 15, # recycle sockets in time_wait state 'net.ipv4.tcp_tw_recycle' => 1, # re-use sockets in time_wait state 'net.ipv4.tcp_tw_reuse' => 1, # don't waste cycles timestamping tcp packets 'net.ipv4.tcp_timestamps' => 0, # higher limit for max socket connections (128 is default) 'net.core.somaxconn' => 37000, 'net.core.netdev_max_backlog' => 4096, 'net.ipv4.tcp_max_syn_backlog' => 4096, 'net.ipv4.tcp_keepalive_time' => 60, 'net.ipv4.tcp_keepalive_probes' => 3 'net.ipv4.tcp_keepalive_intvl' => 90, 'net.core.rmem_max' => 8388608, 'net.core.wmem_max' => 8388608 } ### mongodb / metadata.rb depends 'sysctl' ### mongodb / recipes / default.rb include_recipe 'sysctl::apply' supermarket.chef.io/cookbooks/sysctl
12. .co.uk www. ### mongodb / attributes / default.rb include_attribute 'ulimit' override['ulimit']['users'] = { node['mongodb']['owner'] => { 'core_limit' => 0, 'filehandle_limit' => 20000, 'process_limit' => 10000 } } ### mongodb / metadata.rb depends 'ulimit' ### mongodb / recipes / default.rb include_recipe 'ulimit' supermarket.chef.io/cookbooks/ulimit
13. .co.uk www. ### mongodb/recipes/default.rb include_recipe "#{cookbook_name}::repo" include_recipe "#{cookbook_name}::sysctl" package 'mongodb' do package_name 'mongodb-org' version node['mongodb']['version'] end template "/etc/init/mongodb.conf" do ### ubuntu upstart init file source "upstart-mongodb.conf.erb" owner 'root' group 'root' mode '0644' variables({ dbpath: node['mongodb']['dbpath'], logpath: File.dirname(node['mongodb']['logfile']), owner: node['mongodb']['owner'], group: node['mongodb']['group'], daemon: node['mongodb']['daemon'], cfgfile: node['mongodb']['configfile'] }) end chef tricks
14. .co.uk www. if node['mongodb']['keyfile']['file'] if node['mongodb']['keyfile']['content'] keydata = node['mongodb']['keyfile']['content'] elsif node['mongodb']['keyfile']['databag'] mongodata = data_bag_item node['mongodb']['keyfile']['databag'], 'data' keydata = mongodata['key'] else Chef::Log.fatal 'MongoDB keyfile specified but no key data is available' end directory File.dirname(node['mongodb']['keyfile']['file']) do owner node['mongodb']['owner'] group node['mongodb']['group'] mode '0755' end file node['mongodb']['keyfile']['file'] do owner node['mongodb']['owner'] group node['mongodb']['group'] mode '0400' backup false content keydata end end chef trick
15. .co.uk www. template node['mongodb']['configfile'] do source 'mongodb.conf.erb' owner 'root' group 'root' mode '0644' variables({ mongos: node['mongodb']['mongos'], dbpath: node['mongodb']['dbpath'], logfile: node['mongodb']['logfile'], port: node['mongodb']['port'], auth: node['mongodb']['auth'], keyfile: node['mongodb']['keyfile']['file'], journal: node['mongodb']['journal'], lock_file: node['mongodb']['logrotate']['lock_file'], extra: node['mongodb']['extra'] }) notifies(:restart, "service[mongodb]") if node['mongodb']['config-restart'] end service 'mongodb' do provider Chef::Provider::Service::Upstart supports status: true, restart: true action [ :enable, :start ] end include_recipe "#{cookbook_name}::logrotate" chef trick
16. .co.uk www. ### mongodb/templates/mongodb.conf.erb # http://docs.mongodb.org/manual/reference/configuration-options/ # Where to store the data. # Note: if you run mongodb as a non-root user (recommended) you may # need to create and set permissions for this directory manually, # e.g., if the parent directory isn't mutable by the mongodb user. <%= @dbpath ? "dbpath=#{@dbpath}" : '' %> # where to log logpath=<%= @logfile %> # ... # Disables write-ahead journaling # nojournal = true <%- if @journal -%> journal=true <%- elsif @journal == false -%> nojournal=true <%- end %> <%- unless @mongos %> smallfiles=true <%- end %>
17. .co.uk www. template node['mongodb']['configfile'] do source 'mongodb.conf.erb' owner 'root' group 'root' mode '0644' variables({ mongos: node['mongodb']['mongos'], dbpath: node['mongodb']['dbpath'], logfile: node['mongodb']['logfile'], port: node['mongodb']['port'], auth: node['mongodb']['auth'], keyfile: node['mongodb']['keyfile']['file'], journal: node['mongodb']['journal'], lock_file: node['mongodb']['logrotate']['lock_file'], extra: node['mongodb']['extra'] }) notifies(:restart, "service[mongodb]") if node['mongodb']['config-restart'] end ### ... mongodb/templates/mongodb.conf.erb # Disable the HTTP interface (Defaults to localhost:28017). nohttpinterface = true # Turns off server-side scripting. This will result in greatly limited # functionality #noscripting = true # Turns off table scans. Any query that would do a table scan fails. #notablescan = true # Disable data file preallocation. #noprealloc = true # Specify .ns file size for new databases. # nssize = <size> <%= @extra %>
18. .co.uk www. ### mongodb/recipes/shard.rb include_recipe 'ebs-raid' # <== this creates Raid EBS volumes! replicaset_name = node['mongodb']['replicaset'] node.default['mongodb']['port'] = 27018 node.default['mongodb']['extra'] = <<-EOF replSet=#{replicaset_name} EOF include_recipe cookbook_name chef trick - shard (replica member)
19. .co.uk www. ### mongodb/recipes/arbiter.rb replicaset_name = node['mongodb']['replicaset'] node.default['mongodb']['port'] = 27018 node.default['mongodb']['journal'] = false node.default['mongodb']['extra'] = <<-EOF replSet=#{replicaset_name} EOF include_recipe cookbook_name chef trick - arbiter
20. .co.uk www. ### mongodb/recipes/mongos.rb # once set, config_servers line must never change on a mongos if not node['mongodb']['config_servers'] config_servers = search( :node, "recipes:#{cookbook_name}\\:\\:configserver" ).map { |n| n['fqdn'] }.join(',') node.set['mongodb']['config_servers'] = config_servers end node.default['mongodb']['mongos'] = true node.default['mongodb']['auth'] = nil node.default['mongodb']['dbpath'] = nil node.default['mongodb']['journal'] = nil node.default['mongodb']['daemon'] = node['mongodb']['mongos_daemon'] node.default['mongodb']['extra'] = <<-EOF configdb = #{node['mongodb']['config_servers']} EOF include_recipe cookbook_name chef trick - mongoS server
21. .co.uk www. ### roles/db-c0.rb name "db-c0" description "mongodb cluster config server c0" run_list [ "role[base]", "role[mongodb-configserver]" ] ### roles/db-s0m0.rb name "db-s0m0" description "mongodb cluster shard s0 m0" override_attributes \ mongodb: { replicaset: "graph-s0" } run_list [ "role[base]", "role[mongodb-shard]" ] chef roles ### roles/mongodb-shard.rb name "mongodb-shard" run_list [ "recipe[mongodb::shard]", "recipe[munin]", "recipe[mms-agent::remove]", "recipe[stackdriver::remove]" ]
22. .co.uk www.
23. .co.uk www. compose.io/articles/debunking-myth-of-raid-10-as-best-practice-on-aws
24. .co.uk www. ◦ RAID 10 w/ 4 x 1000 piops drives avg 13.4 ops/sec, 16.1 ops/sec max ◦ Single 4000 piops drive avg 21.4 ops/sec, 23.4 ops/sec max ◦ RAID 10 w/ 4 x 4000 piops drives avg 38.3 ops/sec, 43 ops/sec max EBS provisoned iops compose.io/articles/debunking-myth-of-raid-10-as-best-practice-on-aws blog.celingest.com/en/2013/02/01/benchmarking-mongodb-replica-aws/
25. .co.uk www. Mongo MMAPv1 smallfiles & ext4 # format a block device if it has not been formatted before # use 'small' usage-type for ext4 (for mongod smallFiles setting) mkfs -t ext4 -T small "/dev/${ebs_device}" # add mount definition if it has not beed added before echo "/dev/${ebs_device} /data ext4 defaults,auto,noatime,noexec 0 0" >> /etc/fstab storage.mmapv1.smallFiles Type: boolean Default: False When true, MongoDB uses a smaller default file size. The storage.mmapv1.smallFiles option reduces the initial size for data files and limits the maximum size to 512 megabytes. storage.mmapv1.smallFiles also reduces the size of each journal file from 1 gigabyte to 128 megabytes. Use storage.mmapv1.smallFiles if you have a large number of databases that each holds a small quantity of data. The storage.mmapv1.smallFiles option can lead the mongod instance to create a large number of files, which can affect performance for larger databases.
26. .co.uk www. Mongo MMAPv1 readahead buffer ebs_readahead_kb=32 ebs_device=xvdf blockdev --setra $ebs_readahead_kb "/dev/${ebs_device}" # persist across reboot echo 'ACTION=="add", KERNEL=="'$ebs_device'", ATTR{bdi/read_ahead_kb} ="'$ebs_readahead_kb'"' \ > /etc/udev/rules.d/85-ebs.rules For the MMAPv1 storage engine: • Ensure that readahead settings for the block devices that store the database files are appropriate. For random access use patterns, set low readahead values. A readahead of 32 (16 kB) often works well. • For a standard block device, you can run sudo blockdev --report to get the readahead settings and sudo blockdev -- setra <value> <device> to change the readahead settings. Refer to your specific operating system manual for more information.
27. .co.uk www. Sharding
28. .co.uk www. Sharding
29. .co.uk www. Sharding
30. .co.uk www. Shard Balancing
31. .co.uk www.
32. .co.uk www. Indexing
33. .co.uk www. Indexing notablescan Specify whether all queries must use indexes. If 1, MongoDB will not execute queries that require a collection scan and will return an error.
34. .co.uk www.
35. .co.il www. Thank you! We invite you to join Operations UK Facebook group on dvps.me/OpsUK www.devopspro.co.uk
36. .co.il www. Reference cloudcraft.co speakerdeck.com/jamestyj/automate-production-ready-mongodb-deployments-and-more slideshare.net/AmazonWebServices/scaling-mongodb-on-amazon-web-services-dat209-aws-reinvent-2013-28436939 speakerdeck.com/nathenharvey/the-joy-of-cooking-deploying-mongodb-with-chef-number-mongochicago