DRBD Tutorial | Setup Linux Disk Replication | CentOS 8

In this article I will share Step-by-Step DRBD 9.0 Installation and Configuration Guide on CentOS 8 using KVM based Virtual Machines. Below topics will be covered in this DRBD Tutorial

• DRBD Installation
• Configure global DRBD configuration
• Create DRBD Resource and Device
• Configure Linux Disk Replication
• Manage DRBD primary and secondary nodes

Overview on DRBD

From LINBIT’s DRBD 9.0 User Guide

DRBD (Distributed Replicated Block Device) is a software-based, shared-nothing, replicated storage solution mirroring the content of block devices (hard disks, partitions, logical volumes etc.) between hosts.

DRBD mirrors data:

• **in real time:**Replication occurs continuously while applications modify the data on the device.
• transparently: Applications need not be aware that the data is stored on multiple hosts.
• synchronously or **asynchronously:**With synchronous mirroring, applications are notified of write completions after the writes have been carried out on all (connected) hosts. With asynchronous mirroring, applications are notified of write completions when the writes have completed locally, which usually is before they have propagated to the other hosts

DRBD support for Red Hat and CentOS

• DRBD is provided officially by Linbit. You can get a free version of DRBD software but to get support, you must have a valid subscription with Linbit
• Red Hat does not provide DRBD package with their Red Hat Network but they do officially support DRBD if you have a valid subscription from Linbit
• For CentOS environment you can use ELPO repo to get the DRBD packages which is an open source community
• For this DRBD Tutorial we will use CentOS 8 with DRBD 9.0 to setup Linux Disk Replication
• Later we will use use this DRBD configuration with our pacemaker 2.0 cluster

Lab Environment

• I am using the same Lab Environment from my previous article where I configured my KVM HA Cluster using Pacemaker 2.0 on RHEL and CentOS 8
• I will not include the steps to install KVM and setup KVM High Availability Cluster in this DRBD tutorial as that would just make it very long
• So I will assume that you have a working High Availability Cluster to install and configure DRBD
• If you only intend to setup Linux Disk Replication using DRBD 9.0 then you do not need a Cluster

Below is my Lab Environment

Configure Lower Level Device for DRBD

To setup Linux Disk Replication using DRBD configuration in my DRBD Tutorial, I have added a new disk /dev/sdb of 5GB to all of my Cluster Nodes

[root@centos8-2 ~]# lsblk
NAME                   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda                      8:0    0  20G  0 disk
├─sda1                   8:1    0   1G  0 part /boot
└─sda2                   8:2    0  19G  0 part
  ├─cl_centos8--2-root 253:0    0  17G  0 lvm  /
  └─cl_centos8--2-swap 253:1    0   2G  0 lvm  [SWAP]
sdb                      8:16   0   5G  0 disk

This will become the lower-level device for your DRBD resource. You may use any type of block device found on your system for this purpose. Typical examples include:

• A hard drive partition (or a full physical hard drive),
• A software RAID device,
• An LVM Logical Volume or any other block device configured by the Linux device-mapper infrastructure,
• Any other block device type found on your system.

In our DRBD tutorial we will go ahead with Logical Volume solution for DRBD configuration

Create a new Physical Volume using /dev/sdb

[root@centos8-2 ~]# pvcreate /dev/sdb
  Physical volume "/dev/sdb" successfully created.

Currently there is no free space available in my Volume group

[root@centos8-2 ~]# vgs
  VG           #PV #LV #SN Attr   VSize   VFree
  cl_centos8-2   1   2   0 wz--n- <19.00g    0

I already have a Volume Group, so I will extend the Volume Group by adding the new Physical Volume on all the three Cluster nodes

[root@centos8-2 ~]# vgextend cl_centos8-2 /dev/sdb
  Volume group "cl_centos8-2" successfully extended

Next I will create a logical volume for my DRBD configuration to setup Linux disk replication across Cluster nodes

[root@centos8-2 ~]# lvcreate -L 4G -n drdb-1 cl_centos8-2
  Logical volume "drdb-1" created.

/dev/cl_centos8-2/drdb-1
/dev/cl_centos8-3/drdb-1
/dev/cl_centos8-4/drdb-1

Install DRBD 9.0 from ELREPO

• By default ELREPO will not be available on CentOS server.
• To install ELREPO, first you should import the GPG key or else the GPG check will fail later.
• Since my Cluster nodes are in private network with no internet access, I have created an offline repository using the elrepo packages.
• This repository is shared over my private network using NFS server
• You can directly go ahead and install elpo-release rpm on all you servers where you plan to setup Linux disk replication

[root@repo-server ~]# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

Next install the ELPO repo for CentOS 8 release

[root@repo-server ~]# rpm -Uvh  https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm
Retrieving https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:elrepo-release-8.1-1.el8.elrepo  ################################# [100%]

For RHEL environment, you can create a repo file /etc/yum.repos.d/linbit.repo to reflect the following changes.

[drbd-9.0]
name=DRBD 9.0
baseurl=http://packages.linbit.com/<hash>/yum/rhel7/drbd-9.0/<arch>
gpgcheck=0

Once the elpo-release rpm is installed on all your Cluster nodes, use dnf to lookup drbd package

[root@centos8-2 ~]# dnf search drbd
============================================================== Name & Summary Matched: drbd ===============================================================
kmod-drbd90.x86_64 : drbd90 kernel module(s)
collectd-drbd.x86_64 : DRBD plugin for collectd
drbd90-utils.x86_64 : Management utilities for DRBD
drbd90-utils-sysvinit.x86_64 : The SysV initscript to manage the DRBD.
drbdlinks.noarch : Program for managing links into a DRBD shared partition

So there are the available drbd package.

We will install kmod-drbd90 and drbd90-utils on all the Cluster nodes

[root@centos8-2 ~]# dnf install kmod-drbd90.x86_64 drbd90-utils.x86_64 -y

DRBD configuration

All aspects of DRBD are controlled in its configuration file, /etc/drbd.conf. Normally, this configuration file is just a skeleton with the following contents:

[root@centos8-2 ~]# cat /etc/drbd.conf
# You can find an example in  /usr/share/doc/drbd.../drbd.conf.example

include "drbd.d/global_common.conf";
include "drbd.d/*.res";

By convention, /etc/drbd.d/global_common.conf contains the global and common sections of the DRBD configuration, whereas the .res files contain one resource section each.

Sample DRBD Configuration File

You will get a sample DRBD configuration file under /usr/share/doc/packages/drbd. I will attach a copy of this drbd configuration file for reference

drbd.conf.example

Configure global_common.conf

We will have a simple global_common.conf with below entries in our DRBD tutorial:

[root@centos8-2 ~]# cat /etc/drbd.d/global_common.conf
global {
 usage-count no;
}
common {
 net {
  protocol C;
 }
}

usage-count

• The usage-count no line in the global section skips sending a notice to the DRBD team each time a new version of the software is installed in your system.
• You could change it to yes if you want to submit information from your system.
• Alternatively, you could change it to ask if you want to be prompted for a decision each time you do an upgrade.
• Either way, you should know that they use this information for statistical analysis only, and their reports are always available to the public at http://usage.drbd.org/cgi-bin/show_usage.pl.

protocol C

• The protocol C line tells the DRBD resource to use a fully synchronous replication
• This means that local write operations on the node that is functioning as primary are considered completed only after both the local and remote disk writes have been confirmed.
• Thus, if you run into the loss of a single node, that should not lead to any data loss under normal circumstances, unless both nodes (or their storage subsystems) are irreversibly destroyed at the same time.

Copy this global_common.conf file to all the cluster nodes

[root@centos8-2 ~]# scp /etc/drbd.d/global_common.conf centos8-3:/etc/drbd.d/
[root@centos8-2 ~]# scp /etc/drbd.d/global_common.conf centos8-4:/etc/drbd.d/

Create DRBD resource

• A per-resource configuration file is usually named /etc/drbd.d/<resource>.res.
• Any DRBD resource you define must be named by specifying a resource name in the configuration.
• In our DRBD Tutorial we will name our resource as drbd1
• Every resource configuration must also have at least two on host sub-sections, one for every cluster node.
• All other configuration settings are either inherited from the common section (if it exists), or derived from DRBD’s default settings.

Below is my sample DRBD resource configuration file

[root@centos8-2 ~]# cat /etc/drbd.d/drbd1.res
resource drbd1 {
 meta-disk internal;
 device /dev/drbd1;
 net {
  verify-alg sha256;
 }
 on centos8-2.example.com {         <-- hostname must match `uname -n` output
  node-id   0;              <-- Assign a unique node-id
  disk   /dev/cl_centos8-2/drdb-1;  <-- Logical Volume on the provided host
  address  192.168.122.10:7789;     <-- IP Address to be used to connect to the node with port
 }
 on centos8-3.example.com {
  node-id   1;
  disk   /dev/cl_centos8-3/drdb-1;
  address  192.168.122.11:7789;
 }
 on centos8-4.example.com {
  node-id   2;
  disk  /dev/cl_centos8-4/drdb-1;
  address  192.168.122.12:7789;
 }
 connection-mesh {
  hosts centos8-2 centos8-3 centos8-4;
 }
}

• By default DRBD uses port between 7788-7790, we have explicitly defined port number as 7789 for internal communication in our DRBD Tutorial
• We must make sure this port is reachable over firewall
• We have a global section where we provide the values which are common across all the servers
• Our DRBD device resource would be /dev/drbd1 i.e. the logical volumes from all the cluster nodes would use /dev/drbd1 to perform Linux disk replication
• On a Linux server with DRBD, you can see the available devices under /dev/drbd*
• We will use sha256 algorithm for sync verification. You can choose any digest algorithm supported by the kernel crypto API in your system’s kernel configuration

Follow drbd9.X man page to get the complete list of supported arguments. Here,

 on host-name [...]           
                            Define the properties of a resource on a particular host or set of hosts.
                            The host-name argument must match the Linux host name (uname -n).

 node-id value             
                            Defines the unique node identifier for a node in the cluster. 
                            The node-id parameter exists since DRBD 9. Its value ranges from 0 to 16; 
                            there is nodefault.

 disk {[disk] | none}      
                            Define the lower-level block device that DRBD will use for storing the actual data. 
                            While the replicated drbd device is configured, the lower-level device must not be
                            used directly. 

 address [address-family] address:port
                            Defines the address family, address, and port of a connection endpoint.

Create and Enable DRBD Resource

Next we must copy the DRBD resource file to all the cluster nodes

[root@centos8-2 ~]# scp /etc/drbd.d/drbd1.res centos8-3:/etc/drbd.d/
root@centos8-3's password:
drbd1.res                                                                                                                100%  492   291.8KB/s   00:00

[root@centos8-2 ~]# scp /etc/drbd.d/drbd1.res centos8-4:/etc/drbd.d/
root@centos8-4's password:
drbd1.res                                                                                                                100%  492   376.7KB/s   00:00

• After you have completed initial resource configuration as outlined in the previous sections, you can bring up your resource.
• When we installed DRBD earlier, a utility called drbdadm was installed as well
• drbdadm is intended to be used for the administration of DRBD resources, such as our newly configured volume
• The first step of DRBD Tutorial and Linux Disk Replication in starting and bringing a DRBD resource online is to initialize its metadata
• Note that the /var/lib/drbd directory is needed beforehand. If it was not created previously when you installed DRBD, create it manually before proceeding

# mkdir /var/lib/drbd

Each of the following steps must be completed on all the cluster nodes.

[root@centos8-2 ~]# drbdadm create-md drbd1

md_offset 4294963200
al_offset 4294930432
bm_offset 4294799360

Found some data

 ==> This might destroy existing data! <==

Do you want to proceed?
[need to type 'yes' to confirm] yes

initializing activity log
initializing bitmap (128 KB) to all zero
Writing meta data...
New drbd meta data block successfully created.

So our DRBD device /dev/drbd1 is ready. Repeat the same steps on other Cluster nodes for DRBD configuration

[root@centos8-3 ~]# drbdadm create-md drbd1
[root@centos8-4 ~]# drbdadm create-md drbd1

Setup Firewall for DRBD

Since we used custom port 7789 for communication between all the cluster nodes in the DRBD configuration, we must allow port 7789 in the firewall rule on all the cluster nodes to setup Linux disk replication across servers

[root@centos8-3 ~]# firewall-cmd --add-port 7789/tcp --permanent
[root@centos8-3 ~]# firewall-cmd --reload

Enable DRBD Device

The next step consists of drbd configuration includes enabling drbd1 resource in order to finish allocating both disk and network resources for its operation:

[root@centos8-2 ~]# drbdadm up drbd1
[root@centos8-3 ~]# drbdadm up drbd1
[root@centos8-4 ~]# drbdadm up drbd1

Check DRBD Device status

drbdadm status output should now contain information similar to the following related to Linux Disk Replication:

[root@centos8-2 ~]# drbdadm status drbd1
drbd1 role:Secondary
  disk:Inconsistent
  centos8-3.example.com connection:Inconsistent
  centos8-4.example.com connection:Inconsistent

By now, DRBD has successfully allocated both disk and network resources and is ready for operation in our DRBD Tutorial.

Configure DRDB Primary server

• Note that the status of the device shows as “inconsistent” since we haven’t indicated yet which of the DRBD devices will act as a primary device and which one as a secondary device.
• This step must be performed on only one node, only on initial resource configuration, and only on the node you selected as the synchronization source.
• To perform this step, issue this command from the server which you want to act as primary node

[root@centos8-2 ~]# drbdadm primary --force drbd1

• After issuing this command, the initial full synchronization will commence the Linux Disk Replication.
• You will be able to monitor its progress via drbdadm status <resource>.
• It may take some time for drbd synchronization depending on the size of the device.

So our DRBD device will consider centos8-2 as primary and will sync the data of /dev/drbd1 across other KVM nodes

[root@centos8-2 ~]# drbdadm status drbd1
drbd1 role:Primary
  disk:UpToDate
  centos8-3.example.com role:Secondary
    replication:SyncSource peer-disk:Inconsistent done:2.91
  centos8-4.example.com role:Secondary
    replication:SyncSource peer-disk:Inconsistent done:2.88

By now, your DRBD device is fully operational, even before the Linux disk replication has completed. Checking the status after few minutes

[root@centos8-2 ~]# drbdadm status drbd1
drbd1 role:Primary
  disk:UpToDate
  centos8-3.example.com role:Secondary
    peer-disk:UpToDate
  centos8-4.example.com role:Secondary
    peer-disk:UpToDate

You can check the device mapping across all the Cluster nodes

[root@centos8-2 ~]# ls -l /dev/drbd/by-disk/cl_centos8-2/drdb-1
lrwxrwxrwx 1 root root 14 May  2 12:05 /dev/drbd/by-disk/cl_centos8-2/drdb-1 -> ../../../drbd1

[root@centos8-3 ~]# ls -l /dev/drbd/by-disk/cl_centos8-3/drdb-1
lrwxrwxrwx 1 root root 14 May  2 12:06 /dev/drbd/by-disk/cl_centos8-3/drdb-1 -> ../../../drbd1

[root@centos8-4 ~]# ls -l /dev/drbd/by-disk/cl_centos8-4/drdb-1
lrwxrwxrwx 1 root root 14 May  2 12:06 /dev/drbd/by-disk/cl_centos8-4/drdb-1 -> ../../../drbd1

Enabling Dual-primary Node (Optional)

• Dual-primary mode allows a resource to assume the primary role simultaneously on more than one node.
• Dual-primary mode requires that the resource is configured to replicate synchronously (protocol C). Because of this it is latency sensitive, and ill suited for WAN environments.
• Additionally, as both resources are always primary, any interruption in the network between KVM nodes will result in a split-brain.
• In this DRBD Tutorial, we will use single primary node and the steps are only for demonstration
• To enable dual-primary mode, set the allow-two-primaries option to yes in the net section of your resource configuration:

resource <resource>
  net {
    protocol C;
    allow-two-primaries yes;
    fencing resource-and-stonith;
  }

• Next update this configuration file on all the cluster nodes
• If you make this change to an existing resource, as always, synchronize your drbd.conf to the peer,
• Execute below command on all the cluster nodes

# drbdadm adjust <resource>

Create File System on DRBD device

• Next we will create a file system on our DRBD device to be able to perform read write operations.
• I will create ext4 file system on /dev/drbd1
• This step is required on any one of the Cluster nodes and the changes will be replicated across as our DRBD is already running

[root@centos8-2 ~]# mkfs.ext4 /dev/drbd1
mke2fs 1.44.6 (5-Mar-2019)
Discarding device blocks: done
Creating filesystem with 1048503 4k blocks and 262144 inodes
Filesystem UUID: e4f134c7-6ef5-4508-9d11-358a030e19a8
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

Now, the DRBD resource is ready to be used in our DRBD tutorial as usual. You can now mount it on a mount point and start saving files to it.

Create a mount point with same name on all the cluster nodes

[root@centos8-2 ~]# mkdir /share
[root@centos8-3 ~]# mkdir /share
[root@centos8-4 ~]# mkdir /share

Next you can manually mount the share to the mount point on one of the Cluster nodes

[root@centos8-2 ~]# mount /dev/drbd1 /share

[root@centos8-2 ~]# df -h /share/
Filesystem      Size  Used Avail Use% Mounted on
/dev/drbd1      3.9G   16M  3.7G   1% /share

Lastly I hope the steps from this DRBD tutorial to install configure DRBD and setup Linux disk replication on CentOS 8 Linux was helpful. So, let me know your suggestions and feedback using the comment section.

References:
Linbit: Introduction to DRBD