LowEndBox - Cheap VPS, Hosting and Dedicated Server Deals

How to Setup a Highly Available Wordpress Site From Scratch, Part 4

How to Setup a Highly Available WordPress Site From Scratch, Part 4In this tutorial series, we are setting up a highly available WordPress web site from scratch.

Part 1 – Introduction, Considerations, and Architecture
Part 2 – Setting Up the VPSes
Part 3 – Setting Up MariaDB Multi-Master Replication
Part 4 – File Replication and Setting Up DRBD
Part 5 – Setting Up OCFS2
Part 6 – Round-Robin DNS, Let’s Encrypt, & Conclusion

File Replication Options

There are a galaxy of file replication technologies.  We’re going to use DRBD but let’s first cover why we didn’t choose some others.  Our requirements are that the replication is both bi-directional and real-time:

rsync: not bi-directional, not real-time.  If you are the only author and the only person who will be uploading media, etc. you might just run rsync after each update (or set it up in a cron job).  This is not a bad option but you’ll need to handle the certbot verification a different way.  I recommend either using DNS validation or reading this article on certbot in RRDNS configurations.

lsyncd: real-time, but not bi-directional.  Only supports one-way syncing.

unison: bi-directional but not not real-time.  You could run unison in a cron job every few minutes on both nodes to keep your nodes in sync if you can live with the lag.

glusterfs: – bi-directional (multi-directional, actually) and real-time, but requires a minimum of three nodes.

mirror: doesn’t preserve owner, group, etc.

In this tutorial, we’ll use DRBD which is both bi-directional and real-time.  Because we want each node to be capable of writing/changing files, we can’t use ext4 or any other “normal” filesystem but instead need a filesystem that supports shared-disk operations.  For our purposes, we’ll use OFCS2 (Oracle Cluster Filesystem 2)

Preparing for DRBD

To use DRBD, we need to have a raw, unformatted partition on each side available.  DRBD works a the block level, not the filesystem level.  In Linode’s manager, I’ve added a new 10GB block storage disk to each node.  If you’re using KVM, you will need to set aside an unused partition when you setup the node.

Start by installing needed packages on both nodes:

apt-get -y install drbd-utils ocfs2-tools

Load the DRBD kernel module on both nodes:

root@web1:~/mysql# modprobe drbd
root@web1:~/mysql# lsmod | grep drbd
drbd                  421888  0
lru_cache              16384  1 drbd
libcrc32c              16384  1 drbd

I checked dmesg to discover that the block storage I’d added was mapped to /dev/sdc:

[17461.350617] scsi 1:0:0:0: Direct-Access     Linode   Volume           2.5+ PQ: 0 ANSI: 5
[17461.353223] sd 1:0:0:0: Power-on or device reset occurred
[17461.355751] sd 1:0:0:0: [sdc] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB)
[17461.357007] sd 1:0:0:0: [sdc] Write Protect is off
[17461.357771] sd 1:0:0:0: [sdc] Mode Sense: 63 00 00 08
[17461.357912] sd 1:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[17461.360936] sd 1:0:0:0: Attached scsi generic sg2 type 0
[17461.365013] sd 1:0:0:0: [sdc] Attached SCSI disk

On both nodes, I partitioned /dev/sdc to add a single primary partition that took up the whole disk:

root@web1:/etc# fdisk /dev/sdc

Welcome to fdisk (util-linux 2.33.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x477ee7d7.

Command (m for help): p
Disk /dev/sdc: 10 GiB, 10737418240 bytes, 20971520 sectors
Disk model: Volume          
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x477ee7d7

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-20971519, default 2048): <return>
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-20971519, default 20971519): 

Created a new partition 1 of type 'Linux' and of size 10 GiB.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

The partition table now looks like this:

root@web1:/etc# fdisk /dev/sdc
Welcome to fdisk (util-linux 2.33.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): p
Disk /dev/sdc: 10 GiB, 10737418240 bytes, 20971520 sectors
Disk model: Volume          
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x477ee7d7

Device     Boot Start      End  Sectors Size Id Type
/dev/sdc1        2048 20971519 20969472  10G 83 Linux

Setting up DRBD

Now let’s setup some configuration files.  On each node, rename /etc/drbd.d/global_common.conf to /etc/drbd.d/global_common.conf.dist and replace it with this:

global {
  usage-count yes;
}

common {
  net {
    protocol C;
  }
}

Protocol C is synchronous replication.  This is the best choice in our scenario because (a) the servers are in the same datacenter so there is reduced latency, and (b) WordPress is a “few writers, many readers” kind of site, where most operations are read rather than write.

Next, create /etc/drbd.d/r0.res and populate it as follows:

resource r0 {
        on web1.lowend.party {
            address 1.1.1.1:7788;
            device /dev/drbd0;
            meta-disk internal;
            disk /dev/sdc1;
        }
        on web2.lowend.party {
            address 2.2.2.2:7788;
            device /dev/drbd0;
            meta-disk internal;
            disk /dev/sdc1;
        }
#        startup {
#        become-primary-on both; 
#        }
        net {
            cram-hmac-alg sha1;
            shared-secret "secret";
            # allow-two-primaries yes;
            after-sb-0pri discard-zero-changes;
            after-sb-1pri discard-secondary;
            after-sb-2pri disconnect;
        }
}

A couple important notes:

  • The “on” clauses must match the output of the ‘hostname’ command.
  • You’ll notice both the “startup” clause and the “allow-two-primaries” part of the “net” clause are commented out.  This is intentional.  We will enable these directives once we have DRBD initialized.

Now on each node, execute these commands:

drbdadm create-md r0
drbdadm up r0

You can now look at /proc/drbd to see the status, or run drbdadm:

root@web1:/etc/drbd.d# cat /proc/drbd 
version: 8.4.10 (api:1/proto:86-101)
srcversion: 983FCB77F30137D4E127B83 
 0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
    ns:0 nr:0 dw:0 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:10484380
root@web1:/etc/drbd.d# drbdadm status r0
r0 role:Secondary
  disk:Inconsistent
  peer role:Secondary
    replication:Established peer-disk:Inconsistent

This output tells use that this volume is in an inconsistent state and both sides of the cluster are secondaries.  In essence, both sides are waiting for the primary to update, but neither is primary so nothing is happening.  Let’s fix that.  On web1, execute this command:

drbdadm primary --force r0

Now if you look at /proc/drbd you’ll see that synchronization is beginning:

root@web1:/etc/drbd.d# cat /proc/drbd 
version: 8.4.10 (api:1/proto:86-101)
srcversion: 983FCB77F30137D4E127B83 
 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
    ns:1732 nr:0 dw:0 dr:1732 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:10482648
 [>....................] sync'ed:  0.1% (10236/10236)M
 finish: 1:27:21 speed: 1,732 (1,732) K/sec

Don’t freak out by the estimate 1 hour, 27 minute sync time for a 10G volume.  I checked /proc/drbd about 10 seconds later and it looked like this:

root@web1:/etc/drbd.d# cat /proc/drbd 
version: 8.4.10 (api:1/proto:86-101)
srcversion: 983FCB77F30137D4E127B83 
 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
    ns:596272 nr:0 dw:0 dr:596272 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:9888108
 [>...................] sync'ed:  5.8% (9656/10236)M
 finish: 0:07:16 speed: 22,640 (14,904) K/sec

Once the volumes are in sync, you’ll see this:

root@web1:/etc/drbd.d# cat /proc/drbd 
version: 8.4.10 (api:1/proto:86-101)
srcversion: 983FCB77F30137D4E127B83 
 0: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
    ns:134792 nr:0 dw:134792 dr:1469 al:120 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

Now change /etc/drbd.d/r0.res on both nodes and uncomment out the commented portions.  You can do this manually or by running the sed command below (be sure to do this on both nodes):

# systemctl stop drbd
# sed -i 's/#//' /etc/drbd.d/r0.res
# systemctl start drbd
# systemctl enable drbd
drbd.service is not a native service, redirecting to systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable drbd

DRBD should now report that it’s both up-to-date (consistent on all nodes) and that all nodes are primaries:

root@web1:/etc# cat /proc/drbd 
version: 8.4.10 (api:1/proto:86-101)
srcversion: 983FCB77F30137D4E127B83 
 0: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
    ns:609047 nr:7596 dw:15911 dr:623182 al:4 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
root@web1:/etc# drbdadm status all
r0 role:Primary
  disk:UpToDate
  peer role:Primary
    replication:Established peer-disk:UpToDate

Our DRBD configuration is now complete.

Next Part: Part 5 – Setting Up OCFS2

 

raindog308

1 Comment

  1. Very interesting article. Thank you.

    January 7, 2021 @ 11:43 pm | Reply

Leave a Reply

Some notes on commenting on LowEndBox:

  • Do not use LowEndBox for support issues. Go to your hosting provider and issue a ticket there. Coming here saying "my VPS is down, what do I do?!" will only have your comments removed.
  • Akismet is used for spam detection. Some comments may be held temporarily for manual approval.
  • Use <pre>...</pre> to quote the output from your terminal/console, or consider using a pastebin service.

Your email address will not be published. Required fields are marked *