Slide 1

Slide 1 text

.co.uk www. MongoDB & Chef Operations Automation Right?

Slide 2

Slide 2 text

.co.uk www. Evgeny Zislis Co-Founder / CTO www.devops.co.il www.devopspro.co.uk

Slide 3

Slide 3 text

.co.uk www.

Slide 4

Slide 4 text

.co.uk www.

Slide 5

Slide 5 text

.co.uk www. mongodb-is-web-scale.com

Slide 6

Slide 6 text

.co.uk www.

Slide 7

Slide 7 text

.co.uk www.

Slide 8

Slide 8 text

.co.uk www.

Slide 9

Slide 9 text

.co.uk www. github.com/chef-brigade/mongodb-cookbook github.com/sunggun-yu/chef-mongodb3

Slide 10

Slide 10 text

.co.uk www. ### mongodb/metadata.rb depends 'apt' ### mongodb/recipes/repo.rb include_recipe 'apt' apt_repository 'mongodb-org' do uri 'http://repo.mongodb.org/apt/ubuntu' distribution "#{node['lsb']['codename']}/mongodb-org/3.2" components [ 'multiverse' ] keyserver 'keyserver.ubuntu.com' key 'EA312927' action :add end docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/ github.com/sunggun-yu/chef-mongodb3/blob/master/recipes/package_repo.rb

Slide 11

Slide 11 text

.co.uk www. ### mongodb / attributes / default.rb include_attribute 'sysctl' override['sysctl']['params'] = { # more ports for open connections - 37k # 27017 is MongoDB listen port! 'net.ipv4.ip_local_port_range' => '28000 65000', # more concurrent connections when spammed 'net.ipv4.tcp_fin_timeout' => 15, # recycle sockets in time_wait state 'net.ipv4.tcp_tw_recycle' => 1, # re-use sockets in time_wait state 'net.ipv4.tcp_tw_reuse' => 1, # don't waste cycles timestamping tcp packets 'net.ipv4.tcp_timestamps' => 0, # higher limit for max socket connections (128 is default) 'net.core.somaxconn' => 37000, 'net.core.netdev_max_backlog' => 4096, 'net.ipv4.tcp_max_syn_backlog' => 4096, 'net.ipv4.tcp_keepalive_time' => 60, 'net.ipv4.tcp_keepalive_probes' => 3 'net.ipv4.tcp_keepalive_intvl' => 90, 'net.core.rmem_max' => 8388608, 'net.core.wmem_max' => 8388608 } ### mongodb / metadata.rb depends 'sysctl' ### mongodb / recipes / default.rb include_recipe 'sysctl::apply' supermarket.chef.io/cookbooks/sysctl

Slide 12

Slide 12 text

.co.uk www. ### mongodb / attributes / default.rb include_attribute 'ulimit' override['ulimit']['users'] = { node['mongodb']['owner'] => { 'core_limit' => 0, 'filehandle_limit' => 20000, 'process_limit' => 10000 } } ### mongodb / metadata.rb depends 'ulimit' ### mongodb / recipes / default.rb include_recipe 'ulimit' supermarket.chef.io/cookbooks/ulimit

Slide 13

Slide 13 text

.co.uk www. ### mongodb/recipes/default.rb include_recipe "#{cookbook_name}::repo" include_recipe "#{cookbook_name}::sysctl" package 'mongodb' do package_name 'mongodb-org' version node['mongodb']['version'] end template "/etc/init/mongodb.conf" do ### ubuntu upstart init file source "upstart-mongodb.conf.erb" owner 'root' group 'root' mode '0644' variables({ dbpath: node['mongodb']['dbpath'], logpath: File.dirname(node['mongodb']['logfile']), owner: node['mongodb']['owner'], group: node['mongodb']['group'], daemon: node['mongodb']['daemon'], cfgfile: node['mongodb']['configfile'] }) end chef tricks

Slide 14

Slide 14 text

.co.uk www. if node['mongodb']['keyfile']['file'] if node['mongodb']['keyfile']['content'] keydata = node['mongodb']['keyfile']['content'] elsif node['mongodb']['keyfile']['databag'] mongodata = data_bag_item node['mongodb']['keyfile']['databag'], 'data' keydata = mongodata['key'] else Chef::Log.fatal 'MongoDB keyfile specified but no key data is available' end directory File.dirname(node['mongodb']['keyfile']['file']) do owner node['mongodb']['owner'] group node['mongodb']['group'] mode '0755' end file node['mongodb']['keyfile']['file'] do owner node['mongodb']['owner'] group node['mongodb']['group'] mode '0400' backup false content keydata end end chef trick

Slide 15

Slide 15 text

.co.uk www. template node['mongodb']['configfile'] do source 'mongodb.conf.erb' owner 'root' group 'root' mode '0644' variables({ mongos: node['mongodb']['mongos'], dbpath: node['mongodb']['dbpath'], logfile: node['mongodb']['logfile'], port: node['mongodb']['port'], auth: node['mongodb']['auth'], keyfile: node['mongodb']['keyfile']['file'], journal: node['mongodb']['journal'], lock_file: node['mongodb']['logrotate']['lock_file'], extra: node['mongodb']['extra'] }) notifies(:restart, "service[mongodb]") if node['mongodb']['config-restart'] end service 'mongodb' do provider Chef::Provider::Service::Upstart supports status: true, restart: true action [ :enable, :start ] end include_recipe "#{cookbook_name}::logrotate" chef trick

Slide 16

Slide 16 text

.co.uk www. ### mongodb/templates/mongodb.conf.erb # http://docs.mongodb.org/manual/reference/configuration-options/ # Where to store the data. # Note: if you run mongodb as a non-root user (recommended) you may # need to create and set permissions for this directory manually, # e.g., if the parent directory isn't mutable by the mongodb user. <%= @dbpath ? "dbpath=#{@dbpath}" : '' %> # where to log logpath=<%= @logfile %> # ... # Disables write-ahead journaling # nojournal = true <%- if @journal -%> journal=true <%- elsif @journal == false -%> nojournal=true <%- end %> <%- unless @mongos %> smallfiles=true <%- end %>

Slide 17

Slide 17 text

.co.uk www. template node['mongodb']['configfile'] do source 'mongodb.conf.erb' owner 'root' group 'root' mode '0644' variables({ mongos: node['mongodb']['mongos'], dbpath: node['mongodb']['dbpath'], logfile: node['mongodb']['logfile'], port: node['mongodb']['port'], auth: node['mongodb']['auth'], keyfile: node['mongodb']['keyfile']['file'], journal: node['mongodb']['journal'], lock_file: node['mongodb']['logrotate']['lock_file'], extra: node['mongodb']['extra'] }) notifies(:restart, "service[mongodb]") if node['mongodb']['config-restart'] end ### ... mongodb/templates/mongodb.conf.erb # Disable the HTTP interface (Defaults to localhost:28017). nohttpinterface = true # Turns off server-side scripting. This will result in greatly limited # functionality #noscripting = true # Turns off table scans. Any query that would do a table scan fails. #notablescan = true # Disable data file preallocation. #noprealloc = true # Specify .ns file size for new databases. # nssize = <%= @extra %>

Slide 18

Slide 18 text

.co.uk www. ### mongodb/recipes/shard.rb include_recipe 'ebs-raid' # <== this creates Raid EBS volumes! replicaset_name = node['mongodb']['replicaset'] node.default['mongodb']['port'] = 27018 node.default['mongodb']['extra'] = <<-EOF replSet=#{replicaset_name} EOF include_recipe cookbook_name chef trick - shard (replica member)

Slide 19

Slide 19 text

.co.uk www. ### mongodb/recipes/arbiter.rb replicaset_name = node['mongodb']['replicaset'] node.default['mongodb']['port'] = 27018 node.default['mongodb']['journal'] = false node.default['mongodb']['extra'] = <<-EOF replSet=#{replicaset_name} EOF include_recipe cookbook_name chef trick - arbiter

Slide 20

Slide 20 text

.co.uk www. ### mongodb/recipes/mongos.rb # once set, config_servers line must never change on a mongos if not node['mongodb']['config_servers'] config_servers = search( :node, "recipes:#{cookbook_name}\\:\\:configserver" ).map { |n| n['fqdn'] }.join(',') node.set['mongodb']['config_servers'] = config_servers end node.default['mongodb']['mongos'] = true node.default['mongodb']['auth'] = nil node.default['mongodb']['dbpath'] = nil node.default['mongodb']['journal'] = nil node.default['mongodb']['daemon'] = node['mongodb']['mongos_daemon'] node.default['mongodb']['extra'] = <<-EOF configdb = #{node['mongodb']['config_servers']} EOF include_recipe cookbook_name chef trick - mongoS server

Slide 21

Slide 21 text

.co.uk www. ### roles/db-c0.rb name "db-c0" description "mongodb cluster config server c0" run_list [ "role[base]", "role[mongodb-configserver]" ] ### roles/db-s0m0.rb name "db-s0m0" description "mongodb cluster shard s0 m0" override_attributes \ mongodb: { replicaset: "graph-s0" } run_list [ "role[base]", "role[mongodb-shard]" ] chef roles ### roles/mongodb-shard.rb name "mongodb-shard" run_list [ "recipe[mongodb::shard]", "recipe[munin]", "recipe[mms-agent::remove]", "recipe[stackdriver::remove]" ]

Slide 22

Slide 22 text

.co.uk www.

Slide 23

Slide 23 text

.co.uk www. compose.io/articles/debunking-myth-of-raid-10-as-best-practice-on-aws

Slide 24

Slide 24 text

.co.uk www. ○ RAID 10 w/ 4 x 1000 piops drives avg 13.4 ops/sec, 16.1 ops/sec max ○ Single 4000 piops drive avg 21.4 ops/sec, 23.4 ops/sec max ○ RAID 10 w/ 4 x 4000 piops drives avg 38.3 ops/sec, 43 ops/sec max EBS provisoned iops compose.io/articles/debunking-myth-of-raid-10-as-best-practice-on-aws blog.celingest.com/en/2013/02/01/benchmarking-mongodb-replica-aws/

Slide 25

Slide 25 text

.co.uk www. Mongo MMAPv1 smallfiles & ext4 # format a block device if it has not been formatted before # use 'small' usage-type for ext4 (for mongod smallFiles setting) mkfs -t ext4 -T small "/dev/${ebs_device}" # add mount definition if it has not beed added before echo "/dev/${ebs_device} /data ext4 defaults,auto,noatime,noexec 0 0" >> /etc/fstab storage.mmapv1.smallFiles Type: boolean Default: False When true, MongoDB uses a smaller default file size. The storage.mmapv1.smallFiles option reduces the initial size for data files and limits the maximum size to 512 megabytes. storage.mmapv1.smallFiles also reduces the size of each journal file from 1 gigabyte to 128 megabytes. Use storage.mmapv1.smallFiles if you have a large number of databases that each holds a small quantity of data. The storage.mmapv1.smallFiles option can lead the mongod instance to create a large number of files, which can affect performance for larger databases.

Slide 26

Slide 26 text

.co.uk www. Mongo MMAPv1 readahead buffer ebs_readahead_kb=32 ebs_device=xvdf blockdev --setra $ebs_readahead_kb "/dev/${ebs_device}" # persist across reboot echo 'ACTION=="add", KERNEL=="'$ebs_device'", ATTR{bdi/read_ahead_kb} ="'$ebs_readahead_kb'"' \ > /etc/udev/rules.d/85-ebs.rules For the MMAPv1 storage engine: ● Ensure that readahead settings for the block devices that store the database files are appropriate. For random access use patterns, set low readahead values. A readahead of 32 (16 kB) often works well. ● For a standard block device, you can run sudo blockdev --report to get the readahead settings and sudo blockdev -- setra to change the readahead settings. Refer to your specific operating system manual for more information.

Slide 27

Slide 27 text

.co.uk www. Sharding

Slide 28

Slide 28 text

.co.uk www. Sharding

Slide 29

Slide 29 text

.co.uk www. Sharding

Slide 30

Slide 30 text

.co.uk www. Shard Balancing

Slide 31

Slide 31 text

.co.uk www.

Slide 32

Slide 32 text

.co.uk www. Indexing

Slide 33

Slide 33 text

.co.uk www. Indexing notablescan Specify whether all queries must use indexes. If 1, MongoDB will not execute queries that require a collection scan and will return an error.

Slide 34

Slide 34 text

.co.uk www.

Slide 35

Slide 35 text

.co.il www. Thank you! We invite you to join Operations UK Facebook group on dvps.me/OpsUK www.devopspro.co.uk

Slide 36

Slide 36 text

.co.il www. Reference cloudcraft.co speakerdeck.com/jamestyj/automate-production-ready-mongodb-deployments-and-more slideshare.net/AmazonWebServices/scaling-mongodb-on-amazon-web-services-dat209-aws-reinvent-2013-28436939 speakerdeck.com/nathenharvey/the-joy-of-cooking-deploying-mongodb-with-chef-number-mongochicago