Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Replicating MongoDB
Search
Philipp Krenn
November 15, 2014
Programming
0
200
Replicating MongoDB
...what could go wrong?
Presentation for the Javantura conference in Zagreb.
Philipp Krenn
November 15, 2014
Tweet
Share
More Decks by Philipp Krenn
See All by Philipp Krenn
Full-Text Search Explained
xeraa
11
2.2k
360° Monitoring of Your Microservices
xeraa
7
3.4k
Scale Your Metrics with Elasticsearch
xeraa
4
140
YAML Considered Harmful
xeraa
0
2.1k
Scale Your Elasticsearch Cluster
xeraa
1
310
Hands-On ModSecurity and Logging
xeraa
2
190
Centralized Logging Patterns
xeraa
1
1.1k
Dashboards for Your Management with Kibana Canvas
xeraa
1
470
Make Your Data FABulous
xeraa
3
890
Other Decks in Programming
See All in Programming
GPUを計算資源として使おう!
primenumber
1
100
A full stack side project webapp all in Kotlin (KotlinConf 2025)
dankim
0
120
Node-RED を(HTTP で)つなげる MCP サーバーを作ってみた
highu
0
120
猫と暮らす Google Nest Cam生活🐈 / WebRTC with Google Nest Cam
yutailang0119
0
140
Result型で“失敗”を型にするPHPコードの書き方
kajitack
5
920
Discover Metal 4
rei315
2
140
PostgreSQLのRow Level SecurityをPHPのORMで扱う Eloquent vs Doctrine #phpcon #track2
77web
2
530
GitHub Copilot and GitHub Codespaces Hands-on
ymd65536
2
150
#QiitaBash MCPのセキュリティ
ryosukedtomita
1
1.4k
なぜ適用するか、移行して理解するClean Architecture 〜構造を超えて設計を継承する〜 / Why Apply, Migrate and Understand Clean Architecture - Inherit Design Beyond Structure
seike460
PRO
3
780
システム成長を止めない!本番無停止テーブル移行の全貌
sakawe_ee
1
210
プロダクト志向ってなんなんだろうね
righttouch
PRO
0
190
Featured
See All Featured
Building Adaptive Systems
keathley
43
2.7k
Building Applications with DynamoDB
mza
95
6.5k
Testing 201, or: Great Expectations
jmmastey
43
7.6k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
48
2.9k
The Art of Programming - Codeland 2020
erikaheidi
54
13k
Adopting Sorbet at Scale
ufuk
77
9.5k
RailsConf 2023
tenderlove
30
1.1k
Measuring & Analyzing Core Web Vitals
bluesmoon
7
510
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.7k
Why Our Code Smells
bkeepers
PRO
336
57k
Gamification - CAS2011
davidbonilla
81
5.4k
Transcript
MongoDB Replication
Philipp Krenn @xeraa ecosio & ViennaDB
Motivation Availability & data safety Read scalability Helping backups
Data migration Delayed members Oplog Tailing (Meteor.js) https://meteorhacks.com/mongodb-oplog-and-meteor.html
Basics
Terminology Primary + Secondaries Master + Slaves problematic — renamed
Arbiter
http://docs.mongodb.org
http://docs.mongodb.org
http://docs.mongodb.org > rs.addArb("arbiter.example.com:3000")
http://docs.mongodb.org
Limits 50 replica set members 12 before 2.7.8 7 voting
members
Example
None
Single instance $ mkdir 1 $ mongod --dbpath 1 --port
27001 --logpath log1 $ mongo --port 27001 > db.test.insert({ name: "Philipp", city: "Wien" }) > db.test.find() Stop instance
Add replication $ mkdir 2 $ mkdir 3 $ mongod
--replSet javantura --dbpath 1 --port 27001 --logpath log1 --oplogSize 20 $ mongod --replSet javantura --dbpath 2 --port 27002 --logpath log2 --oplogSize 20 $ mongod --replSet javantura --dbpath 3 --port 27003 --logpath log3 --oplogSize 20
Connect $ hostname $ mongo --port 27001 > db.test.find()
Configure replication Start on the old instance, otherwise data lost
rs.initiate() rs.status() rs.add("PK-MBP:27002") rs.add("PK-MBP:27003") rs.status() db.isMaster() db.test.find() db.test.insert({ name: "Peter", city: "Steyr" }) db.test.find()
Read from secondaries $ mongo --port 27002 > db.test.find() >
rs.slaveOk() > db.test.find() > db.test.insert({ name: "Dieter", city: "Graz" }) slaveOk only valid for the current connection
Failover Kill primary with [Ctrl]+[C] Write to new primary >
rs.status() > db.test.insert({ name: "Dieter", city: "Graz" }) > db.test.find()
Restart old primary $ mongod --replSet name --dbpath 1 --port
27001 --logpath log1 --oplogSize 20 $ mongo --port 27001 > rs.status() > rs.slaveOk() > db.test.find()
Election
Heartbeat 2s interval 10s until election
Election rules 1. Priority 2. Optime 3. Connections
Priority cfg = rs.conf() cfg.members[0].priority = 0 cfg.members[1].priority = 1
cfg.members[2].priority = 2 rs.reconfig(cfg)
Optime
Connections
Election Candidate node asks for a vote Others can veto
Election One yes for one node within 30s Majority yes
elects a new primary
None
Issues
CAP Select Availability or Consistency Partition-tolerance is a prerequisite for
distributed systems "The network is reliable": http://aphyr.com/posts/288-the-network-is-reliable
Rollback Old primary rolls back unreplicated changes once it rejoins
the replica set
Rollback file rollback/ in data folder File name: <database>.<collection>. <timestamp>.bson
Election time At times 5 to 7 minutes http://www.tokutek.com/2014/07/explaining-ark- part-2-how-elections-and-failover-currently-work/
Missing synchronization during election Old primary sends last changes to
a single node If not new primary: rollback
Remember Replication is asynchronous
Multiple primaries Unlikely but possible Bugs: https://jira.mongodb.org/browse/SERVER-9765 Test script with
no replies: https://groups.google.com/ forum/#!topic/mongodb-dev/-mH6BOYyzeI
Kyle Kingsbury @aphyr: Call Me Maybe http://aphyr.com/tags/jepsen PostgreSQL, Redis, MongoDB,
Riak, Zookeeper, RabbitMQ, etcd + Consul, ElasticSearch
http://aphyr.com/posts/284-call-me- maybe-mongodb 05/2013 version 2.4 Up to 42% data lost
Data written to old primary: rollback
None
WriteConcern Configure durability vs performance https://github.com/mongodb/mongo-java-driver/blob/ master/src/main/com/mongodb/WriteConcern.java
WriteConcern. UNACKNOWLEDGED w=0, j=0 Fire and forget Default until 11/2012
None
WriteConcern. ACKNOWLEDGED w=1, j=0 Current default Operation successful in memory
WriteConcern. JOURNALED w=1, j=1 Operation written to the journal file
Since 1.8, single server durability
WriteConcern.FSYNCED w=1, fsync=true Operation written to disk
WriteConcern. REPLICA_ACKNOWLEDGED w=2, j=0 Acknowledged by primary and at least
one secondary w is the server number
WriteConcern. MAJORITY w=majority, j=0 Acknowledgement by the majority of nodes
wtimeout recommended
WriteConcern. MAJORITY Nearly no data lost, but high overhead
Write concern performance https://blog.serverdensity.com/mongodb-on-google- compute-engine-tips-and-benchmarks/ 3 x 1,000 inserts on
GCE Local 10GB system disk Dedicated 200GB disk Dedicated 200GB for data and journal
n1-standard-2
n1-highmem-8
Thanks! Questions? Now, later today, or @xeraa
Backup Slides
Oplog
Replication via logs MongoDB: Operations log (Oplog) MySQL: Binary log
(Binlog)
Naiv approach: Transmit original query Statement Based Replication (SBR) DELETE
FROM test.table WHERE quantity > 20 LIMIT 1 db.collection.remove({ quantity: { $gt: 20 }}, true) //justOne: true
Unambiguous representation Row-Based Replication (RBR): Oplog
MongoDB Asynchronous replication Secondaries can get the Oplog from: their
primary a secondary with more recent data
Oplog size 32bit: 48MB 64bit OS X: 183MB 64bit *nix,
Windows: 1GB to 50GB (5% free disk)
Inner details
Capped collection in oplog.rs of the local database > use
local > show collections me 0.000MB / 0.008MB oplog.rs 0.000MB / 20.000MB replset.minvalid 0.000MB / 0.008MB slaves 0.000MB / 0.008MB startup_log 0.003MB / 10.000MB system.indexes 0.001MB / 0.008MB system.replset 0.000MB / 0.008MB
> db.oplog.rs.find() { "h": NumberLong("-265486071808715859"), "ns": "test.test", "o": { "_id":
ObjectId("541a8ed285ea5f8ae059d530"), "name": "Dieter" "city": "Graz" }, "op": "i", "ts": Timestamp(1411026642, 1), "v": 2 } ...