DRBD Pierre Mavro www.enovance.com September 11, 2013 

DRBD: Summary Summary 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2 Basic usages and understandings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3 Use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Pierre Mavro www.enovance.com DRBD 2 / 62 

DRBD: Introduction How it works DRBD refers to block devices designed as a building block to form high availability (HA) clusters. This is done by mirroring a whole block device via an assigned network. DRBD can be seen as network based raid-1. Pierre Mavro www.enovance.com DRBD 3 / 62 

DRBD: Introduction Synchronisations DRBD supports three distinct replication modes, allowing three degrees of replication : A (Asynchronous replication) : Master node disk write + local TCP send buffer → operation completed B (Semi synchronous replication) : Master node disk write + replication packets reached other node → operation completed C (Synchronous replication protocol) : Written to both the local and the remote disk to get operation completed The Syncronous replication protocol is the most robust protocol tested in production. http://www.drbd.org/home/mirroring/ Pierre Mavro www.enovance.com DRBD 4 / 62 

DRBD: Introduction Data accessibility A consequence of mirroring data on block device level is that you can access your data (using a file system) only on the active node. This is not a shortcoming of DRBD but is caused by the nature of most file systems (ext3, XFS, JFS, ext4, ...). These file systems are designed for one computer accessing one disk, so they cannot cope with two computers accessing one (virtually) shared disk. In spite of this limitation, there are still a few ways to access the data on the second node: Use DRBD on logical volumes and use LVM’s capabilities to take snapshots on the standby node, and access the data via the snapshot. DRBD’s primary-primary mode with a shared disk file system (GFS, OCFS2). These systems are very sensitive to failures of the replication network. Mount in read only mode the partition We are using DRBD in master/slave at eNovance. Pierre Mavro www.enovance.com DRBD 5 / 62 

DRBD: Basic usages and understandings Plan 2 Basic usages and understandings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Configuration and initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Check replication status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Node switching and manual synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Remove a DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Pierre Mavro www.enovance.com DRBD 6 / 62 

DRBD: Basic usages and understandings Configuration The DRBD configuration should be the same on both nodes. This drbd.conf file is the global configuration of the DRBD service with fine tuning configuration : /etc/drbd.conf global { <-- Global configuration usage-count no; <-- Do not report statistics usage to LinBit } common { <-- All resources inherit the options set in this section protocol C; <-- C (Synchronous replication protocol) startup { wfc-timeout 1 ; <-- Wait for connection timeout (in seconds) degr-wfc-timeout 1 ; <-- Wait for connection timeout, if this node was a degraded } cluster (in seconds) Pierre Mavro www.enovance.com DRBD 7 / 62 

DRBD: Basic usages and understandings Configuration /etc/drbd.conf net { max-buffers 8192; <-- Maximum number of requests to be allocated by DRBD max-epoch-size 8192; <-- The highest number of data blocks between two write barriers sndbuf-size 512k; <-- The size of the TCP socket send buffer unplug-watermark 8192; <-- how often the I/O subsystem’s controller is forced to process pending I/O requests cram-hmac-alg sha1; <-- The HMAC algorithm to enable peer authentication at all shared-secret "xxx"; <-- The shared secret used in peer authentication # Split brains after-sb-0pri disconnect; <-- Split brain, resource is not in the Primary role on any host after-sb-1pri disconnect; <-- Split brain, resource is in the Primary role on one host after-sb-2pri disconnect; <-- Split brain, resource is in the Primary role on both host rr-conflict disconnect; <-- Helps to solve the cases when the outcome of the resync } decision is incompatible with the current role assignment handlers { pri-on-incon-degr "echo node is primary, degraded and the local copy of the data is inconsistent | wall "; <-- If the node is primary, degraded and if the } local copy of the data is inconsistent Pierre Mavro www.enovance.com DRBD 8 / 62 

DRBD: Basic usages and understandings Configuration /etc/drbd.conf disk { on-io-error pass_on; <-- The node downgrades the disk status to inconsistent on io errors no-disk-barrier; <-- Disable protecting data if power failure (done by hardware) no-disk-flushes; <-- Disable the backing device to support disk flushes no-disk-drain; <-- Do not let write requests drain before write requests of a new reordering domain are issued no-md-flushes; <-- Disables the use of disk flushes and barrier BIOs when accessing } the meta data device syncer { rate 300M; <-- The maximum bandwidth a resource uses for background re-synchronization al-extents 3833; <-- Control how big the hot area (= active set) can get } } Pierre Mavro www.enovance.com DRBD 9 / 62 

DRBD: Basic usages and understandings Configuration Now the resources can be defined. One resource should match a DRBD device. Both nodes informations should be filled : /etc/drbd.d/resources.conf resource drbd1 { <-- DRBD block device name syncer {after drbd0 } <-- Start drbd1 after drbd0 is up & running on srv1 { <-- Master node name device /dev/drbd1; <-- Block device name of the resource being described disk /dev/sda1; <-- Block device to store and retrieve the data address x.x.x.x:7789; <-- IP address and port of the local host meta-disk internal; <-- The last part of the backing device is used to store the meta-data } on srv2 { <-- Slave node name device /dev/drbd2; disk /dev/sda1; address y.y.y.y:7789; meta-disk internal; } } Pierre Mavro www.enovance.com DRBD 10 / 62 

DRBD: Basic usages and understandings Initialisation Here is a summary to understand the process to create a drbd replication. You need to follow that rule to get a working DRBD synchronization : Init metadata device Attach DRBD device Connect DRBD device Start syn- chronization Pierre Mavro www.enovance.com DRBD 11 / 62 

DRBD: Basic usages and understandings Init metadata device This step must be completed only on initial device creation. It initializes DRBD’s metadata (replace by the drbd device name you want to initialize). You should complete those steps on both nodes : Create device metadata on server 1 srv1~$ drbdadm create -md Writing meta data ... initialising activity log NOT initializing bitmap New drbd meta data block sucessfully created. Create device metadata on server 2 srv2~$ drbdadm create -md ... Pierre Mavro www.enovance.com DRBD 12 / 62 

DRBD: Basic usages and understandings Attach and connect DRBD device Attach a local backing block device to the DRBD resource’s device : Attach DRBD device $ drbdadm attach If the peer device is already configured, the two DRBD devices will connect : Sets up the network configuration of the resource’s device $ drbdadm connect Pierre Mavro www.enovance.com DRBD 13 / 62 

DRBD: Basic usages and understandings Start synchronization To start the first synchronization, you need to ask to the first server to sync all blocks to the secondary server. This will erase all blocks on the other server, so all data will be lost : Start the first synchronization $ drbdadm -- --overwrite -data -of -peer primary This action may take a while, depending on the network bandwidth and DRBD volume size. Pierre Mavro www.enovance.com DRBD 14 / 62 

DRBD: Basic usages and understandings Plan 2 Basic usages and understandings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Configuration and initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Check replication status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Node switching and manual synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Remove a DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Pierre Mavro www.enovance.com DRBD 15 / 62 

DRBD: Basic usages and understandings Check replication status To check the replication status, simply look at DRBD file in /proc : Check $ cat /proc/drbd version: 8.3.7 (api :88/ proto :86 -91) srcversion: EE47D8BF18AC166BE219757 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/ Inconsistent C r. ns :912248 nr:0 dw:0 dr :920640 al:0 bm :55 lo:1 pe :388 ua :2048 ap:0 [=== >................] sync ’ed: 21.9% (3283604/4194304) K finish: 1:08:24 speed: 580 (452) K/sec Pierre Mavro www.enovance.com DRBD 16 / 62 

DRBD: Basic usages and understandings Check replication status Another solution consists to launch this command that will provide informations of all DRBD devices : Check $ drbd -overview 0: home Connected Primary/Secondary UpToDate/UpToDate C r--- /home xfs 200G 158G 43G 79% Pierre Mavro www.enovance.com DRBD 17 / 62 

DRBD: Basic usages and understandings Check replication status Several informations need be checked to know the status replication of a DRBD device : Replication status Connection state (cs) Resources roles (ro) Disk states (ds) Pierre Mavro www.enovance.com DRBD 18 / 62 

DRBD: Basic usages and understandings Check replication status Connection states (cs) : StandAlone : No network configuration available. The resource has not yet been connected, has been administratively disconnected or has dropped its connection due to failed authentication/split brain. Unconnected : Temporary state, prior to a connection attempt. WFConnection : This node is waiting until the peer node becomes visible on the network. Connected : A DRBD connection has been established, data mirroring is now active. This is the normal state. PausedSync : The local node is the source or target of an ongoing synchronization, but synchronization is currently paused. This may be due to a dependency on the completion of another synchronization process, or due to synchronization having been manually interrupted. Pierre Mavro www.enovance.com DRBD 19 / 62 

DRBD: Basic usages and understandings Check replication status Resource roles (ro) : Primary : The resource is currently in the primary role, and may be read from and written to. This role only occurs on one of the two nodes, unless dual-primary mode is enabled. Secondary : The resource is currently in the secondary role. It normally receives updates from its peer (unless running in disconnected mode), but may neither be read from nor written to. This role may occur on one or both nodes. Unknown : The resource’s role is currently unknown. The local resource role never has this status. It is only displayed for the peer’s resource role, and only in disconnected mode. Pierre Mavro www.enovance.com DRBD 20 / 62 

DRBD: Basic usages and understandings Check replication status Disk states (ds) : UpToDate : Consistent, up-to-date state of the data. This is the normal state. Inconsistent : The data is inconsistent. This status occurs immediately upon creation of a new resource, on both nodes (before the initial full sync). Also, this status is found in one node (the synchronization target) during synchronization. Pierre Mavro www.enovance.com DRBD 21 / 62 

DRBD: Basic usages and understandings Plan 2 Basic usages and understandings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Configuration and initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Check replication status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Node switching and manual synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Remove a DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Pierre Mavro www.enovance.com DRBD 22 / 62 

DRBD: Basic usages and understandings Node switching You may need to perform maintenance tasks. For instance you need to switch a DRBD volume to the other node. Simply connect to the secondary node and launch : Set secondary as primary $ drbdadm primary You can also switch all volumes at once. Double check if you really can do this before proceed : Set secondary as primary $ drbdadm primary all Pierre Mavro www.enovance.com DRBD 23 / 62 

DRBD: Basic usages and understandings Node switching You also can switch a Primary node to Secondary : Set primary as secondary $ drbdadm secondary Notes This will not automatically set the old secondary node to primary state Pierre Mavro www.enovance.com DRBD 24 / 62 

DRBD: Basic usages and understandings Manual synchronization To start a manual synchronization, you need to invalidate the DRBD device on the current host : Invalidate data on current host $ drbdadm invalidate You can do the same from one node to the other node : Invalidate data on remote host $ drbdadm invalidate_remote You can then check the status in /proc/drbd Pierre Mavro www.enovance.com DRBD 25 / 62 

DRBD: Basic usages and understandings Plan 2 Basic usages and understandings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Configuration and initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Check replication status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Node switching and manual synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Remove a DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Pierre Mavro www.enovance.com DRBD 26 / 62 

DRBD: Basic usages and understandings Remove a DRBD device Before removing a DRBD device, be sure you’ve dumped all data ! Then disconnect the device on both nodes : Disconnect DRBD device $ drbdadm disconnect And delete it : Remove DRBD device $ drbdsetup down Change volume_number by the number seen with drbd-overview command. To finish, remove the config files. Pierre Mavro www.enovance.com DRBD 27 / 62 

DRBD: Use cases Plan 3 Use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Stop, upgrade and commit changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Stop, upgrade and rollback changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Up, connect...but doesn’t want to come UpToDate . . . . . . . . . . . . . . . . . . . . . . . 35 How to promote a primary when having a dual secondary . . . . . . . . . . . . . . . . . . 39 Pierre Mavro www.enovance.com DRBD 28 / 62 

DRBD: Use cases Stop, upgrade and commit changes Sometimes, you may need to perform maintenances tasks on a DRBD device. For example, you need to upgrade stuffs on a DRBD device and if success, commit changes to the secondary : DRBD Primary DRBD Secondary Set secondary node in maintenance mode Resume sync Upgrade success Normal state Pierre Mavro www.enovance.com DRBD 29 / 62 

DRBD: Use cases Stop, upgrade and commit changes To set the secondary node in maintenance, simply stop the synchronisation on the secondary node : Disconnect drbd device $ drbdadm disconnect Do what you have to do on the DRBD primary device and once you have finished and everything looks fine, resume the synchronization to get back to the normal state : Connect drbd device $ drbdadm connect Pierre Mavro www.enovance.com DRBD 30 / 62 

DRBD: Use cases Plan 3 Use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Stop, upgrade and commit changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Stop, upgrade and rollback changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Up, connect...but doesn’t want to come UpToDate . . . . . . . . . . . . . . . . . . . . . . . 35 How to promote a primary when having a dual secondary . . . . . . . . . . . . . . . . . . 39 Pierre Mavro www.enovance.com DRBD 31 / 62 

DRBD: Use cases Stop, upgrade and rollback changes Sometimes, you may need to perform maintenances tasks on a DRBD device. For example, you need to upgrade stuffs on a DRBD device and if fail, rollback changes from the secondary : DRBD Primary DRBD Secondary Set secondary node in maintenance mode Upgrade failed Normal state Rollback sync Pierre Mavro www.enovance.com DRBD 32 / 62 

DRBD: Use cases Stop, upgrade and rollback changes To set the secondary node in maintenance, simply stop the synchronisation on the secondary node : Disconnect drbd device $ drbdadm disconnect Then you’ve upgraded and want to rollback. You have to switch your secondary node to master node : Promote device as primary $ drbdadm primary Pierre Mavro www.enovance.com DRBD 33 / 62 

DRBD: Use cases Stop, upgrade and rollback changes Then you need to invalidate the datas on the old primary : Invalidate drbd device data $ drbdadm invalidate To finish, adjust and connect the device to perform the rollback on the new master node : Adjust and connect $ drbdadm adjust $ drbdadm connect Pierre Mavro www.enovance.com DRBD 34 / 62 

DRBD: Use cases Plan 3 Use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Stop, upgrade and commit changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Stop, upgrade and rollback changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Up, connect...but doesn’t want to come UpToDate . . . . . . . . . . . . . . . . . . . . . . . 35 How to promote a primary when having a dual secondary . . . . . . . . . . . . . . . . . . 39 Pierre Mavro www.enovance.com DRBD 35 / 62 

DRBD: Use cases Up, connect...but doesn’t want to come UpToDate If you don’t understand why after several up, connect etc.... you don’t see anything changing and still got this kind of event : Check $ drbd -overview 0: drbd0 StandAlone Primary/Unknown UpToDate/DUnknown r----- You need to instigate more. The first thing to do is to look at the logs : /var/log/syslog [ 6375.849509] block drbd0: Split-Brain detected but unresolved, dropping connection! [ 6375.850025] block drbd0: helper command: /sbin/drbdadm split-brain minor-0 [ 6375.852859] block drbd0: helper command: /sbin/drbdadm split-brain minor-0 exit code 0 (0x0) [ 6375.852867] block drbd0: conn( WFReportParams -> Disconnecting ) [ 6375.852873] block drbd0: error receiving ReportState, l: 4! [ 6375.853274] block drbd0: asender terminated [ 6375.853277] block drbd0: Terminating drbd0_asender [ 6375.853506] block drbd0: Connection closed [ 6375.853514] block drbd0: conn( Disconnecting -> StandAlone ) [ 6375.853523] block drbd0: receiver terminated [ 6375.853526] block drbd0: Terminating drbd0_receiver Pierre Mavro www.enovance.com DRBD 36 / 62 

DRBD: Use cases Up, connect...but doesn’t want to come UpToDate Anyway, it may not be clear enough for you. You can use that command to get a better understanding on the situation : Show all drbd informations $ drbdadm show -gi +--< Current data generation UUID >- | +--< Bitmap ’s base data generation UUID >- | | +--< younger history UUID >- | | | +-< older history >- V V V V B63AABEFCFADD87D :2 C30515E7D05B35D :82 C4146DCB8E6BFF :82 C3146DCB8E6BFF :1:1:1:0:0:0:0 ^ ^ ^ ^ ^ ^ ^ -< Data consistency flag >--+ | | | | | | -< Data was/is currently up -to -date >--+ | | | | | -< Node was/is currently primary >--+ | | | | -< Node was/is currently connected >--+ | | | -< Node was in the progress of setting all bits in the bitmap >--+ | | -< The peer ’s disk was out -dated or inconsistent >--+ | -< This node was a crashed primary , and has not seen its peer since >--+ flags: Primary , StandAlone , UpToDate Pierre Mavro www.enovance.com DRBD 37 / 62 

DRBD: Use cases Up, connect...but doesn’t want to come UpToDate Here the problem is clear : we’re in a split brain situation. So there is no other solutions than doing a full sync. But what if no split brain was detected ? There’s a last solution : Adjust and connect the drbd device $ drbdadm adjust $ drbdadm connect Now you can check the replication which is resuming. Pierre Mavro www.enovance.com DRBD 38 / 62 

DRBD: Use cases Plan 3 Use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Stop, upgrade and commit changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Stop, upgrade and rollback changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Up, connect...but doesn’t want to come UpToDate . . . . . . . . . . . . . . . . . . . . . . . 35 How to promote a primary when having a dual secondary . . . . . . . . . . . . . . . . . . 39 Pierre Mavro www.enovance.com DRBD 39 / 62 

DRBD: Use cases How to promote a primary when having a dual secondary If you get the dual secondary state after node’s restart, check which node as the ’UpToDate’ state : Check $ drbd -overview 0: drbd0 Connected Secondary/Secondary UpToDate/UpToDate C r----- Here, both are up to date. Simply select one and promote it as primary : Set as primary $ drbdadm primary You should now see one primary and a secondary synced. Pierre Mavro www.enovance.com DRBD 40 / 62 

DRBD: Ninja tricks Plan 4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Create DRBD replication without syncing data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 What to do in Split brain case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Mount secondary node as read only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 All is good, but still doesn’t want to connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Can’t "create-md" on an old DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 My DRBD device is still there but not the configuration . . . . . . . . . . . . . . . . . . . 58 Speed up sync transfer rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 Pierre Mavro www.enovance.com DRBD 41 / 62 

DRBD: Ninja tricks Create DRBD replication without syncing data When you have big devices to sync on a DRBD initialization, it could take several hours or days in worst cases. There is a solution to only sync bitmap and have your DRBD device available in a few seconds instead ! Create on both nodes the DRBD device : Create device metadata $ drbdadm create -md Writing meta data ... initialising activity log NOT initializing bitmap New drbd meta data block sucessfully created. Pierre Mavro www.enovance.com DRBD 42 / 62 

DRBD: Ninja tricks Create DRBD replication without syncing data Then bring it up (still on both nodes) : Connect the volume $ drbdadm up Only on the master node, initialize the bitmap and uuid of the DRBD device : Only sync bitmap $ drbdadm -- --clear -bitmap new -current -uuid Pierre Mavro www.enovance.com DRBD 43 / 62 

DRBD: Ninja tricks Create DRBD replication without syncing data Promote this master node as primary node to replicate bitmap : Set primary volume $ drbdadm primary Now you can check the DRBD is ready to use :-) Check $ drbd -overview 1: drbd1 Connected Primary/Secondary UpToDate/UpToDate C r----- Pierre Mavro www.enovance.com DRBD 44 / 62 

DRBD: Ninja tricks Plan 4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Create DRBD replication without syncing data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 What to do in Split brain case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Mount secondary node as read only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 All is good, but still doesn’t want to connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Can’t "create-md" on an old DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 My DRBD device is still there but not the configuration . . . . . . . . . . . . . . . . . . . 58 Speed up sync transfer rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 Pierre Mavro www.enovance.com DRBD 45 / 62 

DRBD: Ninja tricks What to do in Split brain case If you get this kind of problem in /proc/drbd : Set secondary as primary $ cat /proc/drbd primary/unknown that mean that you are in a split-brain case. First of all you need to check the version on both nodes ! You absolutely need to have the same version to avoid errors (8.0-8.3/8.4/9.0) : Set secondary as primary $ drbdadm --version | grep DRBDADM_VERSION = DRBDADM_VERSION =8.3.13 Pierre Mavro www.enovance.com DRBD 46 / 62 

DRBD: Ninja tricks What to do in Split brain case You will need to discard all data on the secondary node and reimport all of them from the master node. Umount all drbd Umount all failed DRBD mounted devices before continuing !!! On the master node, we need to connect all failed DRBD devices first : Connect DRBD $ drbdadm connect Pierre Mavro www.enovance.com DRBD 47 / 62 

DRBD: Ninja tricks What to do in Split brain case And then sync all data to the secondary node : Sync from master $ drbdadm -- --discard -my -data connect Check the sync state in /proc/drbd : Check sync state $ cat /proc/drbd version: 8.3.7 (api :88/ proto :86 -91) srcversion: EE47D8BF18AC166BE219757 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/ Inconsistent C r. ns :912248 nr:0 dw:0 dr :920640 al:0 bm :55 lo:1 pe :388 ua :2048 ap:0 [=== >................] sync ’ed: 21.9% (3283604/4194304) K finish: 1:08:24 speed: 580 (452) K/sec Mount your DRBD device when finished. Pierre Mavro www.enovance.com DRBD 48 / 62 

DRBD: Ninja tricks What to do in Split brain case Note : there is a difference between "invalidate" and "discard-my-data" : drbdadm invalidate : This forces the local device of a pair of connected DRBD devices into SyncTarget state, which means that all data blocks of the device are copied over from the peer. This command will fail if the device is not part of a connected device pair. drbdadm – –discard-my-data : Use this option to manually recover from a split-brain situation. In case you do not have any automatic after-split-brain policies selected, the nodes refuse to connect. By passing this option you make this node a sync target immediately after successful connect. Pierre Mavro www.enovance.com DRBD 49 / 62 

DRBD: Ninja tricks Plan 4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Create DRBD replication without syncing data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 What to do in Split brain case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Mount secondary node as read only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 All is good, but still doesn’t want to connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Can’t "create-md" on an old DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 My DRBD device is still there but not the configuration . . . . . . . . . . . . . . . . . . . 58 Speed up sync transfer rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 Pierre Mavro www.enovance.com DRBD 50 / 62 

DRBD: Ninja tricks Mount secondary node as read only There are several ways to mount secondary node in a read only state but several ways to break your data on the secondary node as well. This method is the best to avoid a full resync. First, create a snapshot on your logical volume on top of the DRBD device : Create logical volume $ lvcreate -s -n -snapshot -L 2G /dev// You’re now able to mount the snapshot as a read only device. To make it work, you absolutely need to specify the filesystem : Mount filesystem $ mount -t ext4 -o ro /dev//-snapshot /mountpoint Backup all your desired data. Pierre Mavro www.enovance.com DRBD 51 / 62 

DRBD: Ninja tricks Mount secondary node as read only You can check any time that your data are still under sync with ’drbd-overview’ command. When you’ve finished, you need to umount the mountpoint : Unmount $ umount /mountpoint and remove the logical volume : Delete LV $ lvremove /dev//-snapshot Pierre Mavro www.enovance.com DRBD 52 / 62 

DRBD: Ninja tricks Plan 4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Create DRBD replication without syncing data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 What to do in Split brain case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Mount secondary node as read only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 All is good, but still doesn’t want to connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Can’t "create-md" on an old DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 My DRBD device is still there but not the configuration . . . . . . . . . . . . . . . . . . . 58 Speed up sync transfer rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 Pierre Mavro www.enovance.com DRBD 53 / 62 

DRBD: Ninja tricks All is good, but still doesn’t want to connect All your configuration is ok, your logs doesn’t give any errors but the connect method doesn’t seam to do something. The last chance is to try the ’adjust’ command : Adjust $ drbdadm adjust $ drbdadm connect This should work now Pierre Mavro www.enovance.com DRBD 54 / 62 

DRBD: Ninja tricks Plan 4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Create DRBD replication without syncing data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 What to do in Split brain case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Mount secondary node as read only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 All is good, but still doesn’t want to connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Can’t "create-md" on an old DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 My DRBD device is still there but not the configuration . . . . . . . . . . . . . . . . . . . 58 Speed up sync transfer rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 Pierre Mavro www.enovance.com DRBD 55 / 62 

DRBD: Ninja tricks Can’t "create-md" on an old DRBD device If you get this kind of error when you want to recreate and md device on an old existing one : Initiate DRBD device $ drbdadm create -md Found some data ==> This might destroy existing data! <== Do you want to proceed? [need to type ’yes ’ to confirm] yes drbdadm create -md nameResource : exited with code 40 To resolve the problem, erase all data on the current device : Flush disk data $ drbdadm wipe -md Now restart the create procedure. Pierre Mavro www.enovance.com DRBD 56 / 62 

DRBD: Ninja tricks Can’t "create-md" on an old DRBD device If it doesn’t work, because you don’t have your configuration any more for example, you can do it manually : Flush disk data $ dd if=/ dev/zero of=/ dev/sdaX bs=1M count =128 Now restart the create procedure. Pierre Mavro www.enovance.com DRBD 57 / 62 

DRBD: Ninja tricks Plan 4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Create DRBD replication without syncing data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 What to do in Split brain case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Mount secondary node as read only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 All is good, but still doesn’t want to connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Can’t "create-md" on an old DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 My DRBD device is still there but not the configuration . . . . . . . . . . . . . . . . . . . 58 Speed up sync transfer rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 Pierre Mavro www.enovance.com DRBD 58 / 62 

DRBD: Ninja tricks My DRBD device is still there but not the configuration You may want to remove the DRBD device but unfortunately you did it in the wrong way or an automatic script didn’t did it well. The solution consist to recreate the configuration file to get the device recovered. Warning Before creating a configuration file in the drbd.d folder, check that you’ve configured the correct device and ports to avoid conflicting with already used DRBD devices. You can check that you’ve recovered your device with drbd-overview. Then you’ll be able to follow the remove procedure. Pierre Mavro www.enovance.com DRBD 59 / 62 

DRBD: Ninja tricks Plan 4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Create DRBD replication without syncing data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 What to do in Split brain case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Mount secondary node as read only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 All is good, but still doesn’t want to connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Can’t "create-md" on an old DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 My DRBD device is still there but not the configuration . . . . . . . . . . . . . . . . . . . 58 Speed up sync transfer rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 Pierre Mavro www.enovance.com DRBD 60 / 62 

DRBD: Ninja tricks Speed up sync transfer rate If you need to speed up the transfer rate because you need to do a full sync between to hosts, you can change it on the fly. It does not make sense to set a synchronization rate higher than the maximum write throughput on your secondary node, so set it up to a correct rate on the secondary node : Set sync rate drbdsetup /dev/ syncer -r 900M Then, when you’ve finished and want to revert to the default value : Set sync rate drbdadm adjust Pierre Mavro www.enovance.com DRBD 61 / 62 

www.enovance.com